Paper:
This paper presents a method to solve the city metro network expansion problem using reinforcement learning (RL). In this method, we formulate the metro expansion as a process of sequential station selection, and design feasibility rules based on the selected station sequence to ensure the reasonable connection patterns of the metro line. Following this formulation, we train an actor critic model to design the next metro line. The actor is a seq2seq network with attention mechanism to generate the parameterized policy which is the probability distribution over feasible stations. The critic is used to estimate the expected reward, which is determined by the output station sequences generated by the actor during training, in order to reduce the training variance. The learning procedure only requires the reward calculation. Thus our general method can be extended to multi-factor cases easily. Considering origin-destination (OD) trips and social equity, we expand the current metro network in Xi'an, China, based on the real mobility information of 24,770,715 mobile phone users in the whole city. The results demonstrate the effectiveness of our method.

Review:
This paper introduces a method for solving the problem of network expansion, in particular, considers the city metro network and its expansion within the new metro line. Since this problem was previously represented as the non-linear integer problem, having an exponential number of constraints or requiring expert knowledge, authors had an initial motivation to appeal to learnable algorithms from the reinforcement learning paradigm. Authors proposed to consider the problem as Markov Decision Process, so that the environment is represented as the city grid, with constructed metro network, before some timestamp conditioned by additional transportation constraints, while each new state in the expansion process of construction one separate metro line is the state of the metro network as graph with new station added to the metro line, considered as the agent’s action. The custom reward function was represented based on metrics like OD trips used in the traditional approaches and the first time used in this problem specific social equity score. The permissibility constraints of action, i.e. adding a new station, were incorporated into policy function, while the attention mechanism was used here as in other seq2seq models, to query the most suitable station among candidates. Authors use the actor-critic approach, widely exploited in reinforcement learning problems, and with all the above-mentioned modification reached the baseline results on the real-world data, obtained from Xi’an city metro network, Shaanxi province, China.
Main arguments 
Apart from the interesting approach, this paper should be rejected due to several reasons: (1) the specific application field, considered in this paper, have to be generalized to more general cases, in order to be valuable for Machine Learning community, (2) from RL algorithms perspective, the novelty of proposed method is questionable due to lack of literature review and advanced approach to deep reinforcement learning, (3) requirement of explainable AI is essential for deployment and in spite of the quite sufficient explanations of the algorithm’s works, this paper does not well justify its superiority over the traditional methods either by theory or practice, due to experiments suffering from the lack of variability and missing the main point of improvement, which leads to generally insufficient contribution (4) the paper is unpolished and lacks specific formulations from the existing literature on related subjects.
 
In detail
The paper does a great job of justifying neither the novelty of its method nor the guarantees of increased performance or efficiency compared to existing methods. Although the proposed method discusses the specific problem and there could be a lack of existing sufficiently good methods in this specific topic, the reinforcement learning paradigm is utilized by authors in a very superficial and general manner. 
Firstly, the paper excludes any comparison to similar problems and already existing methods of RL used by similar fields, which can be expressed as planning with constraints (for example, path planning in mobile robot navigation), thus will be more valuable to ML community than specific application of graph expansion problem, limited to line-by-line addition. 
Secondly, except general notice of combinatorial problems solved by RL algorithms, the literature review lacks mentioning graph representation methods or any seq2seq algorithms (which efficiently uses attention), which should serve as an initial guess for the well-performing model in this particular problem. Regarding the RL approach, there are plenty of already solved planning problems, papers comparing RL algorithms and their performances on grid-based, discrete environment problems in order to mention here and ideally, comparing to proposed one within the same specific problem formulation. Because of this reason, there is no justification of concrete architecture construction and choice of actor-critic model for learning this policy in this specific example and so there is no intuition, why this model will work better than existing methods. Besides, this leads to the doubtful novelty of the proposed algorithm, although used first time in this specific field, but composed totally from the same components and sometimes in the same combination as in other works related to RL methods in planning.
Moreover, the explanation of the methods’ architecture and basic components’ influence on the system is insufficient but might serve as an initial explanation of how this algorithm works. However, the number of experiments and their variations leaves much to be desired to justify its practical advantages and its utilization purposes as explainable AI. (1) there is no ablation study conducted to justify the choice of the proposed model over existing reinforcement learning methods, like actor-critic methods and their several variations with baseline, including modifications like attention model, graph representation, encoder and decoder architecture. The mentioning of these methods in the literature review is enough to answer this issue. (2) Using only one, hardly tuned baseline method on one dataset (without any reference to existing benchmark results on it) does not prove the efficiency and good performance of the model. Additional experiments can on different environments structure (city grids, metro network) of complexity of the environments (the width of the cell in the grid) to measure the performance of the algorithm based on hyperparameters (3) Real-world data brings more application-based matter into the subject, but requires more thorough investigation on the optimality of the solutions, proposed by RL method. This means that the comparison with the groundtruth metro network expansion (built after October 15) cannot be used as it is without using the expert evaluation. As a solution and habit in the machine learning literature, without the expert knowledge on the dataset, authors may propose a comparison of the real-world solution with the algorithm’s results, based on the optimized metrics (OD trips score and social equity) included into reward function. 
 
Finally, the proposed paper is imprecise in several ways. In terms of RL formulation and usage of RL terms (commonly introduced in literature), it reveals inappropriate usage of MDP terms and insufficient description policy-gradient approach in actor-critic training algorithm. While not appealing to this issue, authors missed the opportunity to formulate this problem as discrete MDP with high-dimensional action space and sparse rewards, thus had less opportunity to research different model-based reinforcement learning problems in discrete environments to conduct better model selection and experiments. Besides, the paper lacks details in some parts, like the usage of 3G cellular mobile data, which was not mentioned in this work, although it is emphasized as an essential part if not a significant contribution of this research.  
 
 
Questions to answer
Introduction
	What is the reason by constructing the city metro networks line by line, not iteratively adding stations to the existing network on the grid? This would make your problem closer to general grid-based planning problems. Do traditional methods expand metro networks only line-by-line and can this be there a limitation?
	How does a “good solution” is represented in the literature with traditional approaches? What are the general measures and constraints in practice?
	Is incorporating social equity concern your contribution? Then why does the maximization of OD trips is not enough (there is no mention of the preferability of social equity metric based on the results of experiments)?
	The effectiveness of the method due to experimental results is still questionable.
Related work
	If the main concern of the paper is still the limitation of other methods which use expert knowledge then it is better to state the usage of additional data (development index) in social equity metric as a reward design part in Appendix to justify this reward engineering
	“We believe that RL has the ability to solve the metro expansion problem” is the statement, which should be substituted by extensive literature review on RL methods used for planning with constraints or specifically graph-based expansion methods. 
Problem formulation
	Is the network graph undirected or directed (imprecise inclusion of “direct links”)? How does this the OD trips are measured (no information here or in Appendix A).
	What is the reason behind 4 constraints provided in problem formulation? Are they used or defined in the literature review papers?
 
Method
	RL and deep learning methods included in the literature review should be the reference of choosing one or another component of the algorithm, including the formula for policy and critic update.
	It is essential to use notation based on RL formulation, for example, use r(s_t,a_t) = 0 if s_t  is not terminal state and r(s_t,a_t) = w(Z|G) in order to state the sparsity of the problem. S_t here is Z_t sequence of chosen stations during the episode (line expansion process) and so on.
	Use precise formulation, for example, the proposed P(Z|G) probability distribution is the trajectory distribution and not policy function, which could be denoted as π(z_t | S,G,M(Z_t)) 
	How does the choice of filter design is justified, based on similar works in the transportation field?
	What is the reason by choosing the 1-dimensional CNN layer to represent candidate stations and what is the concrete input information? There is no formal specification of the input data, regarding the existing graph structure, only a short mention of it.
	Is the attention mechanism architecture, used here, similar to other models used in seq2seq problems? (Answer: Yes) What is the motivation to use this concrete architecture? 
	Why does the permissibility mapping using filter is used in policy function? Are there reasons to use it in policy function, rather than include in reward design? For example, the commonly used approach is to penalize unfeasible actions (stations) with a very low reward, during the episode learning?
	Training: 
	How does the sparsity of reward influence the performance? Is there a way to better design reward, so that there will not be a necessity to update actor and critic only after the end of the episode (termination step)?
	What is the reason for learning critic as V(Z) = w(Z|G)? Is there a need for baseline if the state and the sequence of actions have bijective mapping, meaning that one sequence of stations can generate the unique state of the environment – line expansion, and vice versa? The intuition of the baseline is to measure the value of the state as the average among all possible actions, which lead to this state.
	What are the batches B used in the actor-critic training procedure?
	How do we generalize the environment training? Do we need to retrain the reinforcement learning algorithm for each different initial metro city network configuration, or it is generalizable to other grids, using the same weights?
Experiments
	Please, include the training time of the RL algorithm and its inference time, and compare it to the performance of the baseline algorithm, which should be one of the most essential contributions, due to the high time-complexity of traditional methods.
	How does the choice of corridor width influence the performance of the baseline method? 
	Is there the baseline performance on the same city metro network (Xi’an city metro) mentioned in other literature, to directly compare with? This is necessary to fill the gap in justification of proposed results optimality (for the baseline case).
	How does the partial similarity of the 2 times line expansion by RL method and 6 real-world lines of the city metro network can justify the optimality of the proposed method? Can you provide the measurement based on OD trips and social equity? Can you provide a truly optimal solution based on the grid granularity, initial network graph, and constraints, to compare with as an optimal solution?