Paper: MetroGNN: Metro Network Expansion with Deep Reinforcement Learning (ICLR-2024)

Reviewer comment (Theory):
In the "Overall Framework" section, the MDP and RL framework do not have a formalized definition, and the description and implementation details of the overall RL framework is rather unclear.

Author response:
Thank you very much for your positive feedback. We have added a formalized definition to both the **MDP** and the **RL framework**, and clarified some notations in the equations. All the additions are highlighted in red in the revised version, and we hope the following responses can adress your concerns.

Reviewer comment (Writing):
Some of the notations in the equations are not sufficiently clarified. For example, in equation (6), it is better to explicitly state that alpha_{i, j} is the relevance measure, and pinpoint the meaning of i and j.

Author response:
Thank you very much for your positive feedback. We have added a formalized definition to both the **MDP** and the **RL framework**, and clarified some notations in the equations. All the additions are highlighted in red in the revised version, and we hope the following responses can adress your concerns.

Reviewer comment (Experiments):
What is the precise definition of OD trips? How is it obtained from your dataset?

Author response:
**Q1.a. What is the precise definition of OD trips?**  

**Response:** Specifically, if an individual starts a journey at region A, reaches region B and remians in B for at least 15 minutes, this travel event contributes a count of 1 to the OD trips from A to B. Furthermore, if the individual subsequently travels to region C, the entire movement will be considered as two separate trips: one from A to B and another from B to C. We have added this definition in Section A.2.  

**Q1.b. How is the OD trips obtained from our dataset?**  

**Response:** We get the spatio-temporal data from a large number of user movement trajectories from Tencent Map, a Map Service Providers in China, and the OD flow matrix in this paper is generated in the way mentioned in Q1.a. We have added the corresponding information in Experiment section.

Reviewer comment (Writing):
In equation (5), which metro line do n_{+-1} and n_{+-2} refer to?

Author response:
**Q3. In equation (5), which metro line do $n_{\pm 1}$ and $n_{\pm 2}$ refer to?**  

**Response:** The **$n_{\pm 1}$ represents all the terminal stations** within the metro network, including both the originating and terminating stations of each metro line.  
The **$n_{\pm 2}$ represents all the subterminal stations** directly connected to the $n_{\pm 1}$.  
Sorry for any confusion caused by the unclear expression, and we have added the appropriate explanation in Method section. To avoid possible ambiguity, the original expression of $n_{\pm 1}$ and $n_{\pm 2}$ has been changed to $n_j \in \mathcal V_{\pm 1}$ and $n_k\in\mathcal V_{\pm 2}$  in Section A.1 of the revised manuscript.

Reviewer comment (Evaluation):
A major weakness is the fact that the authors do not report aggregated results across several runs together with statistical confidence measures (e.g. error bars). These should be presented for all figures and tables in order to account for the stochasticity of model training. Otherwise, a possible alternative explanation for the observed results is that one of the "lucky" seeds was used, which yields better performance than what we might see in the average case. It is not possible, in my opinion, to draw reliable conclusions from the presented results. In case multiple runs were indeed carried out, these details should be reported.

Author response:
**Response:** Thank you for your suggestions regarding the reliability of our method. In the previous version, we run multiple experiments with different seeds, and only report the best results in the original manuscript. According to your valuable suggestion, in the revised version, **we have integrated the results of multiple (10) runs of the experiment in the Experiment section**, please refer to the Table 2, Figure 3 and Figure 6 in the revised manuscript.

Reviewer comment (Reproducibility):
Lack of (mathematical) precision in the description of the problem and solution method. The authors should aim for a level of clarity that would enable someone to reproduce the results starting from the descriptions only. This is currently not the case.

Author response:
**Response:** We acknowledge the need for clarity in our mathematical formulations, particularly in the definition of the MDP, and we are sorry for the inprecise formulation. We have revised the mathematical expressions of MDP and discussed them in the Method section of the main text with more precise descriptions:

- **State:** The state $S_t$ is a three-tuple $(\mathcal{M}_t, b_t, l_t)$ containing the current metro network $\mathcal{M}_t=(\mathcal{V}_t,\mathcal{E}_t)$, the remaining budget $b_t$, and the number of new metro lines that can be built $l_t$.
- **Action:** The action $A_t$ corresponds to the selection of a single node in $\mathcal{N}$. Considering that the metro network expansion includes the extension of existing lines and the construction of new lines and needs to satisfy corresponding constraints, available actions are defined as:
  $A(S_t=(\mathcal{M}_t, b_t))=\{n \in \mathcal{N} | n \in (\mathcal{X}^e(\mathcal{M}_t,b_t) \cup \mathcal{X}^c(\mathcal{M}_t,b_t,l_t)) \},$
where $\mathcal{X}^e(\mathcal{M}_t,b_t)$ and $\mathcal{X}^c(\mathcal{M}_t,b_t,l_t)$ represent the set of regions that are available for extension and construction based on specific constraints, respectively. The details of diverse constraints are in Section A.1.

Reviewer comment (Experiments):
Potential limited scalability: the authors consider networks with up to 60 nodes, whereas the real Beijing network has approximately ~500 nodes. Hence, the problem as considered is a simplification, and this should be acknowledged. I expect that the observed performance improvement does not come "for free", and that the method suffers in scalability and has substantially longer running times than the classic methods. Studying the scalability of the method (around what number of nodes does it fail to find satisfactory solutions compared to the baseline) and adding representative runtimes for the methods would improve the manuscript.

Author response:
**Response:** We are sorry for making you confused of the graph size. Actually, the graphs for Beijing and Changsha contain a total of **1,166 and 469 nodes**, respectively. It is possible that "B=50" in original Table 1 (Table 2 in revised version) caused this misunderstanding, which represents the expansion budget of $50 billion. We apologize for the ambiguity and have added detailed descriptions of the dataset in Table 1 and provided more precise descriptions.



| City    | Beijing | Changsha |
| ------- | ------- | -------- |
|   Nodes      |   1166      |     469     |
|   Edges      |   4656      |     1642     |
|   Latitude       |  [39.5410,40.2319]       |    [27.9295,28.4276]      |
| Longitude  |    [116.0385,116.8612]     |   [112.7357,113.2599]    |




**Q2.b.  The model has substantially longer running time than classic methods.**

**Response:** Thank you very much for pointing out this important issue. We would like to provide more explanations in this response.
Firstly, **classic methods do not have a significant runtime advantage over our method**. For example, the evolutionary algorithms take an average of **2 hours**, while the mathematical programming methods take **over 3 hours and need additional manual adjustments** for the metro expansion of Beijing network. 
Secondly, those classic methods **can not generate quality and stable solution** for metro network expansion, and **must regenerated from scratch for different metro networks**.
In contrast, though MetroGNN requires 8 hours of training, **it constantly generate optimal and stable solutions**. Moreover, the complementary experiments show that **it takes only 20 seconds** for the well-trained agent to generate better metro expansion solutions for different metro networks compared with DRL-CNN.
According to your valuable comment, we have added the above discussion to Appendix A.5, and please check the revised manuscript for details.

Reviewer comment (Experiments):
The analysis in 4.5 is tenuous, especially given the lack of multiple runs. For example, if we were to stop the process at step 60, we could draw a different conclusion from the one presented in this paragraph. Does this finding repeat across 10+ runs?

Author response:
**Response:** Thank you for your valuable advice. We have added the results with multiple runs to the Figure 6 in the revised manuscript. The results show that the disadvantage of the attentive policy network is just a coincidence. In particular, the model with attentive policy network consistently generates better expansion solutions than the mlp-only policy network in the case of a complete 100 iterations.

Reviewer comment (Novelty):
The technical contributions of this work are limited. While the metro network expansion task is essentially a transportation network combinatorial optimization problem, there have been many existing works [1] studying how to apply RL combined with GNN or attention to address it. It seems the authors only introduce some of the same or similar methods to a specific combinatorial optimization problem. However, there are no substantial technical innovations.

Author response:
**Response:** Thank you for your comment. We would like to summarize the innovations of our work as follows:

- **We propose a specifically designed GNN model for heterogeneous multi-graphs.** This model enables effective capturing of intricate patterns in OD trips which is overlooked by traditional methods.
- **We scale up the metro network more than tenfold compared to existing studies.** The attentive policy network and carefully designed action mask enhances the agent's efficiency in exploring the solution space, enabling our proposed RL model to handle urban region graphs of more than one thousand nodes.
- **Our proposed model exhibits remarkable transferability.** The trained model demonstrates its ability to quickly generate high-quality expansion solutions when faced with different metro networks or varying OD trips. Please refer the response to Q2 of reviewer zGme for more details. This adaptability distinguishes our approach and enhances its applicability across diverse scenarios.

Reviewer comment (Evaluation):
While this work investigates a realistic metro network expansion problem, it only aims to optimize the total satisfied OD flow. However, many other factors need to be considered and optimized to construct a realistic metro network, e.g., social equity or fairness, environmental impact, and revenues. It’s hard to evaluate whether the proposed method is applicable in real scenarios.

Author response:
**Response:** We genuinely appreciate your comment on the problem formulation and completely agree that metro network expansion requires careful consideration of multiple factors. On the one hand, the primary objective of the metro network is to effectively address the travel needs of citizens, thus we believe that the OD flow is a decisive factor in the planning of the metro network. On the other hand, we add additional experiments that incorporate the **social equity** consideration. Specifically, we quantify the inequity as the **population-weighted variance of the shortest distance from regions to the metro network**, and use the decline in inequality for evaluation. We also introduce $\alpha$ and $\beta$ for the adjustment of the weighting between OD flows and equity, respectively. As shown in the following table (higher is better), **our proposed MetroGNN consistently outperforms DRL-CNN under different weightings, achieving better expansion targets with lower variance** (\* denotes to p-value < 0.1, and \*\* denotes to p-value < 0.05).


|       Method        | $\alpha=1.0,\beta=0.0$ |                 |                     | $\alpha=0.5,\beta=0.5$ |                 |                     | $\alpha=0.0,\beta=1.0$ |                 |                    |
|:-------------------:| ---------------------- | --------------- | ------------------- | ---------------------- | --------------- | ------------------- | ---------------------- | --------------- | ------------------ |
|                     | OD                     | equity          | weighted            | OD                     | equity          | weighted            | OD                     | equity          | weighted           |
|       DRL-CNN       | 15.78$\pm$ 1.33        | 12.32$\pm$ 1.46 | 15.78$\pm$ 1.33     | 13.20$\pm$ 1.72        | 20.72$\pm$ 1.94 | 16.96$\pm$ 1.80     | 11.86$\pm$ 2.32        | 26.70$\pm$ 2.16| 26.70$\pm$ 2.16|
|MetroGNN| 18.93$\pm$ 0.87  ** |   11.58$\pm$ 0.26     | 18.93$\pm$ 0.87  ** |      16.50$\pm$ 1.14 **                 |   23.90$\pm$ 1.28     | 20.20$\pm$ 1.21  ** |                 10.49$\pm$ 1.85       |  28.42$\pm$ 1.49  *      | 28.42$\pm$ 1.49  * |
| impr% v.s.  DRL-CNN |            19.96            |   -6.01     | 19.96               |           25.00             |    15.35    | 19.10               |             -11.55           |    6.44    | 6.44               |


Above results have been added in Section A.7.

Reviewer comment (Experiments):
Some important experimental setups are not mentioned or clearly described. For example, the statistics and analysis of datasets. The implementation and hyper-parameter details of baselines. Such information is very significant for the evaluation of experimental reliability.

Author response:
**Response:** Thank you for bringing this to our attention. We have added the details of the dataset at Experiment section, and the implementation of baselines are added at Appendix A.3.

Reviewer comment (Experiments):
The collected real OD flow data are based on the realistic transportation network. If the metro networks have be changed, how can the authors obtain the corresponding OD flow?

Author response:
**Response:** Thank you for this valuable comment. Though there are actually no methods that can predict the OD flow after expansion, research [1] provides an idea for evaluating the feasibility of a metro network to cope with changing OD flows. Following [1], we assume that OD flows do not change drastically after the expansion of the metro network, and add a 5% random noise to all OD trips. As shown in the follow table, **MetroGNN outperforms the DRL-CNN with p-value < 0.003** and allows small variations in OD flow (experiments are repeated for 10 times with different random seeds).



| Method              | 1         | 2         | 3         | 4         | 5         | 6         | 7         | 8         | 9         | 10        |
| ------------------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- |
| MPG                 | 12.71     | 12.49     | 11.87     | 9.18      | 13.54     | 15.77     | 14.53     | 18.80     | 13.61     | 14.86     |
| DRL-CNN             | 15.93     | 14.31     | 15.86     | 15.10     | **16.76** | 15.71     | 16.57     | 17.29     | 16.57     | **17.98** |
| MetroGNN**          | **18.63** | **16.93** | **17.91** | **16.58** | 16.69     | **19.44** | **18.83** | **20.17** | **17.79** | 17.48     |
| impr% v.s.  DRL-CNN | 16.95     | 18.31     | 12.93     | 9.80      | -0.42     | 23.74     | 13.64     | 16.66     | 7.36      | -2.78     |


[1] Wang, L., Jin, J. G., Sibul, G., & Wei, Y. (2023). Designing Metro Network Expansion: Deterministic and Robust Optimization Models. Networks and Spatial Economics, 23(1), 317-347.

The above results are shown in Section A.5 of the revised manuscript.

Reviewer comment (Novelty):
Moreover, using graph neural networks to learn state representation is also not a novel technique. The network proposed in this paper is not well-justified to have sufficient novelty.

Author response:
**Response:** Thank you for your concern on the innovation of our work. We would like to summarize the three major technical innovations of our work as follows:
- **We propose a novel GNN model specially designed for heterogeneous multi-graphs.** It allows us to effectively capture the intricate patterns of OD trips, while traditional methods merely focus on the spatial information of regions. 
- **We scale up the problem to more than ten times larger than those considered in previous studies.** With the attentive policy network and the meticulously designed action mask, the agent can search the solution space much more efficiently, enabling our proposed RL model to handle urban region graphs of more than one thousand nodes. 
- **Our proposed model exhibits excellent transferability.** The trained model demonstrates the ability to swiftly generate high-quality expansion solutions when confronted with different metro networks or varying OD trips. Please refer the response to Q2 for more details. This adaptability sets our approach apart and enhances its utility across diverse scenarios.

Reviewer comment (Novelty):
Using action masks to eliminate infeasible regions is also a common approach in RL applications.

Author response:
**Response:** Thank you for your concern on the innovation of our work. We would like to summarize the three major technical innovations of our work as follows:
- **We propose a novel GNN model specially designed for heterogeneous multi-graphs.** It allows us to effectively capture the intricate patterns of OD trips, while traditional methods merely focus on the spatial information of regions. 
- **We scale up the problem to more than ten times larger than those considered in previous studies.** With the attentive policy network and the meticulously designed action mask, the agent can search the solution space much more efficiently, enabling our proposed RL model to handle urban region graphs of more than one thousand nodes. 
- **Our proposed model exhibits excellent transferability.** The trained model demonstrates the ability to swiftly generate high-quality expansion solutions when confronted with different metro networks or varying OD trips. Please refer the response to Q2 for more details. This adaptability sets our approach apart and enhances its utility across diverse scenarios.

Reviewer comment (Novelty):
Other than insufficient novelty in the algorithm, the model built by this paper is also preliminary. For example, it is natural to consider that more regions could emerge as the city is expanding. The model in this paper is obviously not a high-fidelity model that could be used in real construction.

Author response:
**Response:** Thank you for your constructive comment on the expansion of cities, and we have added a series of experiments to showcase how our model handle the growing urban regions. Firstly, **we remove 20% of the fringe regions of the city to train our model and then applied the trained model to the original complete urban regions**. The following table illustrate the strong transferability of MetroGNN. When directly applied to a larger city, MetroGNN can quickly (within 20 seconds) generate solutions which are comparable to DRL-CNN. **Notably, the fine-tuned model shows an improvement of more than 10.2% over the DRL-CNN** (\* denotes to p-value < 0.1, and \*\* denotes to p-value < 0.05).
| from scratch**  | directly transfer | fine-tune*      | DRL-CNN         |
| --------------- | ------------------ | --------------- | --------------- |
| 18.93 $\pm$ 0.87 | 15.64 $\pm$ 1.25 |  17.39 $\pm$ 1.04   | 15.78 $\pm$ 1.33 |

Secondly, **we introduce 5% noise to OD trips data with 10 different seeds to simulate the changes with urban development**. The following table demonstrates that **MetroGNN can allow variations in OD flow and still outperforms existing approaches**.

| Method              | 1         | 2         | 3         | 4         | 5         | 6         | 7         | 8         | 9         | 10        |
| ------------------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- |
| MPG                 | 12.71     | 12.49     | 11.87     | 9.18      | 13.54     | 15.77     | 14.53     | 18.80     | 13.61     | 14.86     |
| DRL-CNN             | 15.93     | 14.31     | 15.86     | 15.10     | **16.76** | 15.71     | 16.57     | 17.29     | 16.57     | **17.98** |
| MetroGNN**          | **18.63** | **16.93** | **17.91** | **16.58** | 16.69     | **19.44** | **18.83** | **20.17** | **17.79** | 17.48     |
| impr% v.s.  DRL-CNN | 14.49     | 15.48     | 11.45     | 8.93      | -0.42     | 19.19     | 12.00     | 14.28     | 6.86      | -2.86     |

Reviewer comment (Writing):
This paper is not well-written. The details of the model and algorithms are not defined in a clear and mathematical way. The presentation and coherence of this paper could be greatly improved by deleting excessive words and sentences.

Author response:
**Response:** Thank you for your valuable comments on the presentation. We have added clear mathematical description of the MDP and the RL framework in Method section and Appendix A.1. We have also removed unnecessary expressions to make the article more concise. We provide critical descriptions of MDP as follows:

- **State:** The state $S_t$ is a three-tuple $(\mathcal{M}_t, b_t, l_t)$ containing the current metro network $\mathcal{M}_t=(\mathcal{V}_t,\mathcal{E}_t)$, the remaining budget $b_t$, and the number of new metro lines that can be built $l_t$.
- **Action:** The action $A_t$ corresponds to the selection of a single node in $\mathcal{N}$. Considering that the metro network expansion includes the extension of existing lines and the construction of new lines and needs to satisfy corresponding constraints, available actions are defined as:
  $A(S_t=(\mathcal{M}_t, b_t))=\{n \in \mathcal{N} | n \in (\mathcal{X}^e(\mathcal{M}_t,b_t) \cup \mathcal{X}^c(\mathcal{M}_t,b_t,l_t)) \},$
where $\mathcal{X}^e(\mathcal{M}_t,b_t)$ and $\mathcal{X}^c(\mathcal{M}_t,b_t,l_t)$ represent the set of regions that are available for extension and construction based on specific constraints, respectively. The details of diverse constraints are in Section A.1.
- **Reward:** Referring to the definition of total satisfied OD flow in Equation 1, the intermediate reward $R_t$ for action $A_t$ is defined as ${C_{od}(\mathcal M_t)-C_{od}(\mathcal{M}_{t-1})}$.