Paper: Planning with an Ensemble of World Models (ICLR-2024)

Reviewer comment (Novelty):
The second contribution, which is making a number of models for a number of scenarios and using a classifier to learn which model to use for the current scenario during planning, is not novel.

Author response:
> **Scalability and generalizability of city-specific models**: The second contribution, making a number of models for a number of scenarios and using a classifer to learn to use which model for the current scenario during planning, is also not novel...

We wish to emphasize that the city classifier is not a major contribution in this work.  We addresses your above concern under *Scalability and generalizability* of city-specific models in the common rebuttal.

Reviewer comment (Experiments):
The proposed method in the second contribution is not promising due to scalability issues — when there are more cities, more models are required, and the classifier needs to be bigger.

Author response:
> **Scalability and generalizability of city-specific models**: The second contribution, making a number of models for a number of scenarios and using a classifer to learn to use which model for the current scenario during planning, is also not novel...

We wish to emphasize that the city classifier is not a major contribution in this work.  We addresses your above concern under *Scalability and generalizability* of city-specific models in the common rebuttal.

Reviewer comment (Experiments):
The proposed method also has generalization issues — for example, what happens if the vehicle is running in an unseen city.

Author response:
> **Scalability and generalizability of city-specific models**: The second contribution, making a number of models for a number of scenarios and using a classifer to learn to use which model for the current scenario during planning, is also not novel...

We wish to emphasize that the city classifier is not a major contribution in this work.  We addresses your above concern under *Scalability and generalizability* of city-specific models in the common rebuttal.

Reviewer comment (Reproducibility):
Question — Compared to C3, C4 allows the other agents/vehicles to traverse on the ground-truth trajectory without being projected to the road while remaining reactive, but how is this achieved? The details are not clear.

Author response:
> I understand that compared to C3, C4 allows the other agents/vehicles to traverse on the ground-truth trajectory without being projected to the road while remaining reactive. But how? I cannot find the details.

This is quite a simple change. Notable point is that the road graph is built using nodes and edges on the road lane centreline. And, this is different from the agent ground truth trajectory in the recorded log. For C3 simulation, the non-playable agents are projected to the closest road graph node and then IDM logic is employed to navigate. Here, even the parked cars are snapped to the road graph and it moves along on the road graph. For C4, the simple change  we made is to restrict the agent path only on the ground truth trajectory. This way, scenario simulation in C4 appears more like a recorded log. On visible evidence is that this allowed parked cars to be as parked cars in C4.

Reviewer comment (Evaluation):
The paper could be strengthened by discussing various methodologies to assess realism. Incorporating statistical measures such as summary statistics, event counts, time intervals between events, or probability distributions could offer a more comprehensive evaluation. The current approach might inadequately capture infrequent but significant events. It is also possible that the methods employed are not described in sufficient detail to fully convey their scope and impact.

Author response:
We urge the reviewer to read the *realism of simulators* part in the common rebuttal comment. We aim to improve the realism of agent behaviors in a constrained setup. We would like to argue that the agents’ behaviors have limited scope or flexibility in nuPlan C3 evaluation settings. 

We strongly believe that we covered a fair amount of possibilities of agent behaviors optimizing for the city-level behavior modeling. During the rebuttal period, we run the same optimization at the scenario level as shown in *scalability and generalizability* in the common comment. This will further capture more infrequent agent behaviors. However, we hypothesize that we can push the same optimization at the agent-level for more infrequent agent behaviors and bring out more significant events in lines with your comments. This is currently out of scope in this work which we proposed as future works. We agree with your comments on statistical measures. We currently provide the stats of minimum gap between agents as one of the reliable measure to show the difference between our proposed Gym and the Default-Gym in comparison with real-world behavior as in recorded data. We can try to compute more statistical measures and include them in the final paper.

Reviewer comment (Experiments):
It may be worthwhile to look at the individual distributions more closely to investigate the performance of the default behaviors as a result of averaging out multiple modalities; this potentially could be solved by combining GMM/MPC where the GMM models the behavior of the traffic. There is substantial work on stochastic MPC in the context of autonomous driving, including non-gaussian trajectory planning, and insights into how this work applies to the given problem would be interesting. (Possible relevant references: https://ieeexplore.ieee.org/document/9133136 or https://arxiv.org/abs/2002.10999)

Author response:
With the comment “averaging out multiple modalities”, our guess is that we average out many traffic behaviors in each city into a single city-level agent behavior model. You are right in pointing it out. To tackle this, instead of optimizing behavior model for each city, we learn to optimize the agent behaviors into different clusters.  For more details, kindly refer *scalability and generalizability* part in the common rebuttal comment.
We will look into the works and also cite in our works.

Reviewer comment (Writing):
Is "city-driver" as the 2nd listed contribution mainly the objective function? I have the feeling I misunderstood this point.

Author response:
City-Driver is our proposed city-adaptive planner where we repurpose MPC-based PDM planner with a city-specific agent behavior (predictive) model.  This agent behavior model is a parametric IDM model optimized using objective function in Equation 1.

Reviewer comment (Experiments):
Complexity and Scalability: The approach of creating city-specific simulations could be complex and resource-intensive, potentially challenging to scale across numerous cities with distinct driving behaviors.

Author response:
> **Scalability and Generalizability**: a) Complexity and Scalability: The approach of creating city-specific simulations could be complex and resource-intensive, potentially challenging to scale across numerous cities with distinct driving behaviors and b) Generalization:  While city-specific models offer increased accuracy, they might limit the generalizability of the motion planning algorithms. A system trained in one city might not perform as well in another without significant retraining.

We address the above 2 concerns under *Scalability and generalizability of city-specific models* in the common rebuttal.

Reviewer comment (Evaluation):
Generalization: While city-specific models offer increased accuracy, they might limit the generalizability of the motion planning algorithms. A system trained in one city might not perform as well in another without significant retraining.

Author response:
> **Scalability and Generalizability**: a) Complexity and Scalability: The approach of creating city-specific simulations could be complex and resource-intensive, potentially challenging to scale across numerous cities with distinct driving behaviors and b) Generalization:  While city-specific models offer increased accuracy, they might limit the generalizability of the motion planning algorithms. A system trained in one city might not perform as well in another without significant retraining.

We address the above 2 concerns under *Scalability and generalizability of city-specific models* in the common rebuttal.

Reviewer comment (Evaluation):
The nuplan dataset is not well balanced (more data collected in Las Vegas compared to the others). Is this going to affect the results to some extents (e.g., LV driver seems to be better than other cities)?

Author response:
> The nuplan dataset is not well balanced (more data collected in Las Vegas compared to the others). Is this going to affect the results to some extents (e.g., LV driver seems to be better than other cities)?

For all optimization and evaluation in the paper, we fetched the same number of scenarios from each city.  For instance, we use 100 nuPlan test scenarios for every scenario type from each city for evaluation, amounting to 1400 scenarios per city. For optimization, we fetch 1400 nuPlan train scenarios from each city.

Reviewer comment (Evaluation):
While the geolocation would impact the overall driving behaviors, it is quite implicit in some sense. Do the authors have an idea of how to measure this driving behavior distribution shift in a better way?

Author response:
> While the geolocation would impact the overall driving behaviors, it is quite implicit in some sense. Do the authors have an idea of how to measure this driving behavior distribution shift in a better way?

In the common rebuttal comment, we propose to learn clustering of driving behaviors instead of city based clusters. We believe that classifying and plotting the scenarios from cities in these cluster space may give a good idea of driving behavior distribution of the cities.