Paper: OfflineLight: An Offline Reinforcement Learning Model for Traffic Signal Control (ICLR-2024)

Reviewer comment (Theory):
Several math formulas are incorrect. For example, Eq. 1 should be something like $G = E_{\rho_{0}} E_{a_0, a_1,\ldots, \sim \pi}[\sum_{t=0}^\infty \gamma^t r_t]$, where $\rho_0$ is the initial state distribution. Also problematic is the definition of $\pi^*$ two lines after Eq. 2 and the $Q(s,a)$ in Section 4.2.2 (missing the expectation and what is $r^n$?).

Author response:
We apologize for our carelessness and thank you for your attention. There are several typos in the formula. In Eq.1, $\gamma^{n}$ should be $\gamma^{t}$,
  and $r^{n}$ of $Q(s,a)$ in Section 4.2.2 should be $r^{t^{'}}$.

Reviewer comment (Novelty):
The proposed "offline Actor-Critic framework" may not be considered as the contribution of this paper. For example, Eq. 3 in this paper is the same as Eq. 7 in BRAC [1]. Eq. 5 is the same as Eq. 2 in CQL [2].

Author response:
We believe that the main contribution of this paper is the innovation of proposing a new framework. It actually encourages novel approaches and breakthroughs in optimizing and balancing these constraints, which is a significant aspect of our work. Our OfflineLight is an example algorithm for offline Actor-Critic framework, which  lies in integrating and leveraging these known techniques to push the boundaries of performance in TSC.

Reviewer comment (Writing):
There are many typos and inaccurate words/punctuation marks/notations in the paper, making the paper hard to follow.

Author response:
We have carefully reviewed and polished our manuscript, making structural adjustments as per your suggestions.

Reviewer comment (Evaluation):
In Table 1, CQL has similar overall performance but shorter error bars than the proposed method OfflineLight, so it may be overclaim to say that the proposed method "shows more robust performance among offline-RL methods."

Author response:
Our analysis across seven datasets demonstrates that OfflineLight outperforms CQL in five out of these seven datasets. In some cases, the improvements brought by OfflineLight involve significant reductions in Average Travel Time (ATT) by several tens of seconds. While in other datasets, the reduction in ATT may appear marginal (a few seconds), hence it is important to consider the broader impact in urban traffic systems. When translated into real-world implications, these time savings can result in considerable reductions in fuel consumption and subsequent decreases in air pollution.

Reviewer comment (Presentation):
The main paper is not self-contained, the authors may remove/reduce Section 5 and move the results of Section 6.6 & 6.7 onto the main paper.

Author response:
Based on feedback, Sections 5 and 6 will be reduced, with essential results from Sections 6.6 & 6.7 integrated into the main paper for coherence.

Reviewer comment (Reproducibility):
What is the reward function you use for the RL formulation of the TSC problem?

Author response:
The reward function in our decentralized MARL approach focuses on maximizing individual rewards for each traffic signal controller (TSC), as detailed in Equation 2. The metric used is the vehicle queue length (QL), with shorter queues indicating higher efficiency and rewards.

Reviewer comment (Writing):
What this the meaning of this sentence: "the Critic is applied to address overestimated values of the Critic"?

Author response:
Apologies for the misunderstanding due to the inappropriate terminology used. The correct term should be 'overestimated values of the Q value,' not 'overestimated values of the Critic.' We have corrected for this in the revision.

Reviewer comment (Writing):
I don't understand this sentence in Section 4.3.1: "where $\pi_{\theta'}$ is the target policy network by constraining the policy." Wouldn't the "target policy network" being an exponential moving average of the learning policy as in BEAR [3]?

Author response:
Apologies for the lack of clarity in our description within Section 4.3.1 of the 'BEAR' paper, particularly regarding the target policy network. To clarify, the implementation of the target policy network in our paper is indeed similar to that in BEAR. We integrate noise into the learning policy to expand the range of action selection.  Specifically, we implement this by calculating the KL divergence. We will clarify in the revision when the paper is accepted.

Reviewer comment (Writing):
For the sentence "In general, the Actor implementation applies PSR to overfitting to narrow peaks in the value estimate..." Why applying PSR to **overfitting** to narrow peaks?

Author response:
Firstly, the primary function of PSR in our Actor implementation is not to directly address overfitting to narrow peaks in the value estimate, but rather to enhance the policy's robustness and generalization. The reference to ‘narrow peaks’ is in the context of the value function landscape. In some scenarios, the value function might exhibit sharp peaks, indicating a high estimated reward for very specific actions or action sequences. Without appropriate regularization, the Actor might overfit to these narrow peaks, leading to a brittle policy that performs well in the training environment but fails to generalize to slightly different or unseen situations.

Reviewer comment (Novelty):
"Our method provides new insights into the OOD problem of offline RL" --- Could you be more specific about what are the new insights?

Author response:
Our method introduces a unique computational paradigm that simultaneously considers constraints on policy (action) and value, focusing on the trade-off between these two aspects. This approach is particularly insightful for addressing the OOD challenges in offline RL. By balancing the constraints on policy and value, our method effectively mitigates the common pitfalls associated with distributional shifts, which are prevalent in offline RL scenarios. OfflineLight, under this paradigm, has set a new benchmark in traffic signal control (TSC) within offline RL.

Reviewer comment (Reproducibility):
How many seeds do you use in composing Table 1?

Author response:
To ensure a fair and consistent comparison across all experimental runs, we employed the same seed for each iteration of our experiments. Specifically, we conducted a total of 80 rounds of training, with each round followed by a testing phase. Additionally, our code has been made publicly available to facilitate transparency and reproducibility.

Reviewer comment (Novelty):
The novelty is not clear for this work. The offline-AC brings in ideas from policy smooth regularization (PSR), and conservative Q-learning combines these ideas and transfers them into the application of traffic signal control. Though the attempts are reasonable, the novelty is not recognized as a novelty closely related to representation learning.

Author response:
**Response:**

  1. Innovation of Offline-AC: Offline-AC is a computational paradigm within offline RL that emphasizes innovation in the constraints on actions and values. This framework is not only applying existing methods; instead, it actually encourages novel approaches and breakthroughs in optimizing and balancing these constraints, which is a significant aspect of our work. 

  2. Innovation in OfflineLight:Our model, OfflineLight, utilizing established concepts such as PSR and Conservative Q values, achieves SOTA results in offline RL, yielding impressive outcomes. This achievement itself signifies novelty. In parallel, methods such as TD3+BC [1], which combines the existing TD3 and BC approaches, have led to surprising and noteworthy results. Similarly, OfflineLight's innovation lies in integrating and leveraging these known techniques to push the boundaries of performance in TSC.  Moreover, our method can be evidenced in our real implementation at some cities. 

  [1] A Minimalist Approach to Offline Reinforcement Learning.

Reviewer comment (Experiments):
Though the algorithm is well explained, the most crucial part is that the dataset is not well explained. Some details are very critical to judge the quality of this dataset. For example, how many policies did the author use to collect the data? Does this policy train on all datasets? These are the major components of this paper but need to be well discussed.

Author response:
**Response:** 

  1.Data Collection Method: Our approach to data collection prioritizes diversity to enhance the model's generalization capabilities. We used a variety of states and rewards, acknowledging that the optimal choices in existing works are still ambiguous. Our offline dataset comprises interactions generated by different policies, akin to the data buffers in off-policy reinforcement learning methods like Q-learning or DDPG. We engage various RL methods to interact with the CityFlow traffic simulator, recording diverse interaction data. This process encompasses different frameworks, traffic states, and rewards, detailed in Table 3 in Appendix A.2.

  2.States of RL Agents: The RL agents employed in our study were trained from scratch. This approach ensures that the distribution of rewards does not converge prematurely, allowing us to discern the impacts of different actions more effectively. If agents had been well trained beforehand, the reward distribution would have stabilized, making it difficult to assess the relative merits of their actions.

  3.Training on Datasets: Our training was conducted on a limited portion of the available datasets. For example, OfflineLight-JN1 was trained on only 20\% of all data from TSC-OID. Despite this limited training, our model demonstrates robustness, as it can be effectively deployed on various other traffic topologies and flows without extensive retraining.

Reviewer comment (Experiments):
For the dataset, how did the author collect data? Is the policy trained on some other dataset or all the datasets used later? Is the RL agent already well-trained before interacting with the environment? If it is trained on all datasets, then the transferability evaluation is not feasible to evaluate the generalization of this offline-AC performance. Could the author give a very detailed description of this part?

Author response:
**Response:** 

  1.Data Collection Method: Our approach to data collection prioritizes diversity to enhance the model's generalization capabilities. We used a variety of states and rewards, acknowledging that the optimal choices in existing works are still ambiguous. Our offline dataset comprises interactions generated by different policies, akin to the data buffers in off-policy reinforcement learning methods like Q-learning or DDPG. We engage various RL methods to interact with the CityFlow traffic simulator, recording diverse interaction data. This process encompasses different frameworks, traffic states, and rewards, detailed in Table 3 in Appendix A.2.

  2.States of RL Agents: The RL agents employed in our study were trained from scratch. This approach ensures that the distribution of rewards does not converge prematurely, allowing us to discern the impacts of different actions more effectively. If agents had been well trained beforehand, the reward distribution would have stabilized, making it difficult to assess the relative merits of their actions.

  3.Training on Datasets: Our training was conducted on a limited portion of the available datasets. For example, OfflineLight-JN1 was trained on only 20\% of all data from TSC-OID. Despite this limited training, our model demonstrates robustness, as it can be effectively deployed on various other traffic topologies and flows without extensive retraining.

Reviewer comment (Evaluation):
In the appendix Figure 4, the comparison seems not convincing. If the advanced MPLight performs very well on the transferred dataset, then the transfer ratio is also low. Could the author provide another metric to evaluate the performance?

Author response:
**Response:** It is essential to clarify the interpretation of the transfer ratio used in our study. The transfer ratio, defined as $t_{transfer}/t_{train}$ , inversely correlates with performance: The higher the ratio, the lower the performance.
  The MPLight  has shown the best transferability performance. But the comparison demonstrates that advanced MPLight does not outperform OfflineLight in Figure 4. The seemingly low transfer ratio for MPLight does not indicate superior performance but rather the opposite. This is a crucial aspect of our analysis, highlighting the effectiveness of OfflineLight in comparison to MPLight under the given metric. 

  To further solidify our argument about the generalization capabilities of OfflineLight, we plan to include several classic RL methods in our analysis.

Reviewer comment (Theory):
In section 4.2.1, the J(θ) is calculated over n and t. But in the formula, Q is independent of n and t. Should the action and state subscribe with n and t?

Author response:
**Response:** It is crucial to note that the Q value in our formulation indeed corresponds to each time step t in the $n^{\text{th}}$ batch. While in Equation (3), for the sake of simplicity, the Q value is not explicitly subscripted with n and t , the calculations preceding this formulation have taken these variables into account. The decision to present the Q value without direct reference to n and t in the equation was a deliberate choice aimed at simplifying the representation and making the equation more readable. However, this should not be misconstrued as an oversight or a simplification in the actual computational process. We will certainly clarify this in the revision.

Reviewer comment (Experiments):
How could the author conclude the Discussion “Why is the Offline-AC the abstract method?”. To justify this idea, more experiment results should be conducted.

Author response:
**Response:**

  1.Offline-AC (Actor-Critic) should be recognized as a computational paradigm within offline RL, uniquely integrating constraints on actions and values. This paradigm is not merely a specific algorithm or technique but represents a broader conceptual approach.

  2.Methods like CQL and BEAR can be viewed as specific instances of the Offline-AC framework. OfflineLight is also implemented under this paradigm. These examples demonstrate the versatility and applicability of the Offline-AC approach in various contexts.

Reviewer comment (Theory):
OfflineLight treats each traffic signal as an independent RL agent. Since there are multiple agents, it is crucial to clarify the definitions of state, action, and reward in order to comprehend the problem thoroughly. It would be highly beneficial if there were a well-defined problem formulation, possibly following a POMDP framework.

Author response:
We appreciate your suggestion regarding the need of more detailed information about the state definition in our methodology. Our approach is grounded in a decentralized MARL framework, operating within the confines of a POMDP. This is an integral aspect of our research design. In Section 3.3 of our paper, we provided a comprehensive formulation that delineates the essential elements of our approach, particularly as they pertain to the POMDP context. This section is crafted to offer clarity and depth to our methodological framework.

  Each agent in our model concentrates on local traffic information regarding state representation. This includes NV,  NV-segments, QL, EP,  ERV, and TMP for training process.

  All these state variables are integrated into our OfflineLight model. However, in practical applications, we primarily utilize the queue length (QL) as the state input. This decision is based on our assessment of QL's relevance and effectiveness in capturing the traffic state's critical aspects for our model's objectives. This can be evidenced in our real implementation at some cities.

Reviewer comment (Novelty):
OfflineLight does not take into account the interactions between multiple traffic lights. In contrast, many studies in Traffic Signal Control (TSC) have explored such interactions using various techniques like the CTDE framework and graph neural networks. Integrating these approaches could substantially enhance the performance of the proposed model.

Author response:
We acknowledge the potential benefits of integrating approaches that consider interactions between multiple agents.  In our next phase of development, we aim to explore the incorporation of offline multi-agent interactions to further improve the results in TSC. However, the current OfflineLight has the advantages as follows: 

   1. Our OfflineLight has shown superior performance in offline RL scenarios. Notably, it outperforms methods like CoLight and AttendLight (baseline methods in this paper), which do take into account the interactions between various intersections. This outcome indicates that, in certain contexts, the decentralized approach of OfflineLight can be more effective than interaction-centric models. Moreover, OfflineLight is decentralized MARL with individual reward, and the difference of traffic topology has little influence on the final performance.

  2. Comparison with CTDE Methods: We have conducted comparative analyses with methods such as CTDE. Interestingly, OfflineLight exhibited better performance even in these comparisons. MPLight and CoLight are typical CTDE methods, but our OfflineLight performs better. Additionally, a significant advantage of OfflineLight is that it does not require interaction with the environment, which can be a crucial factor in real-world applicability.

Reviewer comment (Reproducibility):
State and Reward Definitions: Could you please provide more information about the definition of the state? Is the state considered a global state or local information for each agent? Additionally, is the reward shared among all agents or individually assigned? This is a critical matter to address, as most RL-based Traffic Signal Control methods operate in a decentralized Multi-Agent Reinforcement Learning (MARL) framework with a Partially Observable Markov Decision Process (POMDP) setting.

Author response:
We appreciate your suggestion regarding the need of more detailed information about the state definition in our methodology. Our approach is grounded in a decentralized MARL framework, operating within the confines of a POMDP. This is an integral aspect of our research design. In Section 3.3 of our paper, we provided a comprehensive formulation that delineates the essential elements of our approach, particularly as they pertain to the POMDP context. This section is crafted to offer clarity and depth to our methodological framework.

  Each agent in our model concentrates on local traffic information regarding state representation. This includes NV,  NV-segments, QL, EP,  ERV, and TMP for training process.

  All these state variables are integrated into our OfflineLight model. However, in practical applications, we primarily utilize the queue length (QL) as the state input. This decision is based on our assessment of QL's relevance and effectiveness in capturing the traffic state's critical aspects for our model's objectives. This can be evidenced in our real implementation at some cities.

Reviewer comment (Experiments):
Offline Dataset Collection Procedure: I'm interested in understanding the specifics of the offline dataset collection procedure. According to appendix B.2, the offline dataset is collected through three epochs of training. However, this may seem insufficient to attain high-reward solutions. Furthermore, I couldn't find information about which RL method is used to generate the dataset for each scenario. Lastly, could you provide some statistics regarding the reward distribution in the dataset, as the quality of the dataset is crucial for Offline RL performance.

Author response:
We appreciate your comment and would like to provide clarifications as follows:

  1. Only Three Epochs of Training:
     (1) Our dataset is unique as it incorporates real-world traffic topology and flow data. This approach ensures that each epoch encompasses a significantly large volume of data. The substantial data volume per epoch compensates for what may initially appear as a limited number of epochs.
     (2) The initial training epochs are particularly crucial for our analysis. In these stages, the reward distribution exhibits a wide range of variability, which is fundamental for algorithmic guidance. This concept is somewhat similar to the Advantage mechanism in Actor-Critic methods like A2C, where the difference between Q-values and value functions is most pronounced in the early stages of learning.

  2. RL Methods Used: To address your query regarding the specific RL methods used for dataset generation in each scenario, we would direct your attention to Appendix Table 5 of our documentation. 

  3. Reward Distribution: We show the distribution of reward of our offline dataset in an anonymous URL : https://anonymous.4open.science/r/OfflineLight_rebuttal-3BA9/README.md.

Reviewer comment (Experiments):
Experiment Section: Can you provide more details about the training and evaluation procedures in your experiments? I'm particularly curious about how the offline RL models are trained and evaluated in the New York scenario, given that there is no available offline dataset. Please elaborate on this aspect.

Author response:
We actually detailed the OfflineLight training process in Appendix A. It can be found that OfflineLight quickly converges in the first three training episodes and shows exceptionally stable and uniformly smooth learning curves across five real-world datasets. One of our main contributions is the generalization of the model. Although OfflineLight-JN1 trained in 20\% of all data from TSC-OID, it can be deployed directly on other traffic topologies and flows. Intuitively pleasing but with little surprise, OfflineLight-JN1 has almost achieved a similar performance to the SOTA RL methods. Even if the New York's topology and flows have not been collected to generate the offline dataset, we can directly deploy to its scenario.  Because the OfflineLight is decentralized MARL with individually reward, the difference of traffic topology has little influence on the final performance.