Paper: MOTSC: Model-based Offline Traffic Signal Control (ICLR-2024)

Reviewer comment (Writing):
There are some minor writing issues that can be improved:  
- p.2: "State: The pressure" -> "State: the pressure" (to be consistent with the next 2 points)  
- p.3: would be good to give the full name of MOPO when it is first introduced (and the abbreviation in the brackets)  
- p.5: "that both forms represents" -> "that both forms represent"  
- p.5: "Given the dataset" -> "Given the datasets"  
- p.5: "resolves the interpolation" -> "resolve the interpolation"  
- p.5: "With assumption 1 and 2" -> "With assumptions 1 and 2"  
- p.7: "Waiting time is the time for one vehicle from stopping to leave the intersection." - maybe "average time" is better?   
- p.8: "online RL method to offline RL" -> "online RL methods to offline RL"  
- p.8: "All the metrics are smaller, the better." -> "The smaller the metrics, the better."  
- p.9: "To have a better understand" -> "To have a better understanding"  
- p.9: "a online method" -> "an online method"  
- p.9: "MOTSC did not it." - it seems that sth is missing here

Author response:
Thanks for your insightful feedback and for helping us find these writing issues, we will fix them as soon as possible.

Reviewer comment (Writing):
On p.5 there is a sentence "Overestimation of the uncertainty happens when u(s, a) does not fully exploit the dataset and prevents the algorithm from exploration.". Could you please explain it? I am not sure if it is clear enough, so might be good to add an explanation (or a reference).

Author response:
Q1: The uncertainty will be added as a penalty to the trained RL agent. When the RL agent wants to explore some unseen states, the trained model will penal this exploration since unlike the real world, the trained transition model does not really know how the unseen state will transit to another state, so we need to add the uncertainty as the penalty to prevent the agent from explore the unseen state unlimitedly. But if we overestimate the uncertainty, the penalty to the agent’s exploration will be too heavy and the agent will be too pessimistic[1].

Hope these can solve your problems.

[1] Why so pessimistic? Estimating uncertainties for offline rl through ensembles, and why their independence matters

Reviewer comment (Experiments):
The paper does not sufficiently explore the extensibility of its core concept, the "movement-independent transition," to alternative applications within offline RL. A more comprehensive examination of its adaptability to diverse offline RL algorithms would enhance the paper's practical utility.

Author response:
Since the “Movement Independent Transition” aims for the TSC problem according to its semantics and mathematical assumptions, it can be only applied to other model-based Offline TSC RL algorithms, and to the best of our knowledge, there is no other Offline RL algorithms aiming to TSC problem appeared in peer-reviewed conferences or journals. But if there are such a kind of algorithms in the future, we can apply them effectively since we first separate the intersection features into the granularity of lanes and do the training normally. Finally, we concatenate the output of the model from lane granularity to intersection granularity for RL agents making decisions.

Reviewer comment (Experiments):
The method uses information corresponding to 12 possible movements at the intersection, and it is not clear whether this is information that can be obtained in a real environment.

Author response:
There are 12 lanes in one intersection (there are four directions, each direction has three lanes for left turn, straight, and right turn) so we can extract 12 movement information from one intersection. Since this is one of our main contributions, we didn’t apply it to other algorithms, but it can be conveniently added to other algorithms as Q1 described, making them perform better.

Reviewer comment (Writing):
Some technical details are not explained clearly. For example, one major problem is the "Concat" in Theorem 1. What does the concat mean mathematically?

Author response:
“Concat” operation is an engineering problem. Since we proposed the concept of “Movement Independent Transition”, we separate the data in one intersection into 12 parts, each standing for a lane to decrease the error in uncertainty estimation while improving the offline data utilization. But for the RL agents, they need the whole intersection features vector as the input, thus we need to “Concat” the output of our model into the input of RL agents.

Reviewer comment (Theory):
Assumptions might be too strict. For example, in assumption 2, the Bayesian neural networks are assumed to be Lipschitz. How to guarantee that in empirical implementations?

Author response:
We followed the assumption in the former work MOPO[1]. In our work, we use the transition model to mimic the real-world traffic flow, which helps the RL agents make better decisions. Since in the real world, the intersection features like queue length and pressure cannot change unlimitedly from time to time, that is why we assume the transition model, i.e. the Bayesian neural networks are K-Lipschitz.

Reviewer comment (Evaluation):
For the empirical evaluation metrics, you learned the offline RL policy on real-world dataset but evaluated the policy using a simulator, which does not look like a fair evaluation. How does the simulation compare to real-world traffic flows? Can you elaborate more on this?

Author response:
The real-world dataset are traffic flow dataset, which presents a vehicle’s trajectory from origin to destination, but how the vehicle interacts with other vehicles, with the environment, and with the traffic light is left to be unknown, we need both real-world dataset and simulator to fully simulate a vehicle’s behavior. Our policy is trained under the real-world traffic flow dataset combined with the simulator which deployed the simplest TSC policy: Fixed time policy. Using these data we have our offline RL agents trained. We evaluate also with real-world traffic flow dataset combined with the simulator but with our trained TSC policy. We can see the improvement compared to the sample policy: Fixed time policy.

Reviewer comment (Novelty):
The traffic signal control problem using deep reinforcement learning is quite an old problem now, with many existing solutions in the literature that incorporate different multi-agent RL (MARL) techniques for a network of TSCs. The current paper only focuses on a single intersection and therefore evades the rich interaction between different neighboring TSCs, which lies at the heart of the problem.

Author response:
Thanks for your insightful feedback.

Q1: This work has two main novelties. First, we study and formulate the problem of TSC under the offline RL setting. Although TSC is an old problem and has many novel methods, it still has a long distance to be implemented in the real world, and one of the main reasons is its unstable performance at the very early stage of online RL training, and what this work wants to do is to some extent make up for this problem. We adapt offline RL methods to TSC problem, using offline dataset train the agent, getting it prepared to be implemented in the real world. Second, we proposed the concept “Movement Independent Transition” and proved its effectiveness mathematically and experimentally in improving the model’s accuracy in estimating the uncertainty, which results in better prediction performance.

Reviewer comment (Experiments):
The techniques should be expanded to multiple TSCs.

Author response:
Thanks for your insightful feedback.

Q2: Actually our method aims to multiple TSCs. All of the experiments shown in Table 1 are conducted under multiple TSCs settings. And the New York setting can even support intersections for up to 192.