Paper: Towards Greener and Sustainable Airside Operations: A Deep Reinforcement Learning Approach to Pushback Rate Control for Mixed-Mode Runways (ICLR-2024)

Reviewer comment (Theory):
In current practice, is the pushback process entirely unmetered? Or would an aircraft need the clearance from air traffic control to pushback? If the latter is true, the ATC would be implicitly doing some optimization right?

Author response:
In most medium to large airports, systems like the Advance Surface Movement Guidance Control System (A-SMGCS) recommends pushback schedules and the final pushback clearances are issued by Air Traffic Controllers (ATCOs). Therefore, in our paper, we also benchmarked our method against non metering baseline which is the schedule based on historical surface movements.

Reviewer comment (Experiments):
Does the methods in the paper use the fact that a runway is mixed use? I'm not sure I see that in the algorithm. Also, for a large airport like Changi, are the runways mixed use?

Author response:
Yes, as per the historical A-SMGCS data analysis, the Changi Airport does operate in mix mode. Operating runways in segregated or mix mode depends on traffic demand, weather, wind conditions etc.  For further clarity, we have modified the section 3 to explicitly highlight the mix mode aspects in the state space formulation. The updated sub-section now reads as follows:

### State Space:
The state observation features capture two crucial types of information: traffic density in hotspot regions and anticipated traffic (both arrivals and departures) demand. While hotspots convey congestion information, traffic demand expresses the expected mix mode runway pressure. Traffic density in hotspots is extracted following the method explained in previous section. A total of six regions (illustrated as polygons in Figure 2 (b)), containing multiple intersections, are identified as potential hotspots. We encode both present and predicted hotspot traffic density for the upcoming time interval (segmented into three periods of two minutes each) across the six hotspots to represent traffic evolution. The second key information about scheduled traffic demand i.e. count of arrivals and departures, expected to be serviced at the mix mode runway in the upcoming time interval (segmented into three periods of two minutes each) is also added to the above hotspot information. To indicate that a runway is mix mode operated, count of arrivals and departures, instead of being summed up, are represented as distinct tensor elements to the DRL agent. Thus, a twenty four dimensional tensor (6X3 + 2X3 = 24) is fed as state observation to the DRL agent.

Reviewer comment (Theory):
Sometimes a plane pushes back to free up a gate such that a landed aircraft can use it. So holding planes at gates is not exactly zero cost.

Author response:
Yes, gate conflicts arising due to departure metering has been studied extensively in previous research Ali et al. 2022; Chen and Solak 2020. To resolve gate conflicts with incoming arrivals, departing aircraft can be towed to metering spots or areas on the apron/taxiways where the aircraft wait until the pushback clearance is given by ATCOs. In this study, taxi delays are calculated from the moment an aircraft receives pushback clearance (starts taxiing) until it takes off from the runway.

**References**

Ali, H., Pham, D. T., Alam, S., & Schultz, M. (2022). Integrated airside landside framework to assess passenger missed connections with airport departure metering. In Proc. Int. Conf. Res. Air Transp. (pp. 1-9).

Chen, H., & Solak, S. (2020). Lower cost departures for airlines: Optimal policies under departure metering. Transportation Research Part C: Emerging Technologies, 111, 531-546.

Reviewer comment (Reproducibility):
Reward shaping and credit assignment are critical in multi-agent settings. How do you ensure that individual agents receive appropriate credit for their actions to promote cooperative behavior? Are there specific reward shaping techniques employed to foster collaborative actions?

Author response:
Since our approach doesn't involve multiple agents, the issues related to reward shaping and credit assignment in a multi-agent setting are not applicable. In this centralized paradigm, the actions are overseen and directed by single DRL agent, and the credit assignment is inherently tied to its decisions in managing the taxiway congestion and runway utilisation.

Reviewer comment (Experiments):
The proposed architecture seems to be extremely tailored for the air traffic control application, which raises concerns about the extreme specificity of the methodology. How generalizable are these methods beyond this application? It would be beneficial to see experiments on a more diverse set of problems.

Author response:
In principle, demand-capacity imbalance problems, encountered in intelligent transportation/traffic management systems, can be tackled by adopting our proposed methodology (or parts thereof). Within Air Traffic Management (ATM) domain, for instance, the event graph idea can be potentially used for Conflict Detection and Resolution (CD-R) in en-route airspace and terminal maneuvering area. We can utilize the timed event graphs in CD-R approaches to accurately predict aircraft trajectories (represented by a sequence of way-points). Besides ATM, Markov Decision Process (MDP) based approaches have shown promise for traffic management in roads (Wei et al., 2021) and warehouses (Zong et al., 2021). Our proposed MDP framework is particularly suited to perform ramp metering where localized on-ramp delays (similar to taxi delays) must be balanced against downstream (bottleneck) freeway utilization (similar to runway utilization). In summary, our proposed method introduce several elements which can be used to solve challenging problems in transportation and logistics.

Reviewer comment (Experiments):
The set of baselines is extremely limited. The authors should provide additional RL-based approaches from the literature or by implementing sensible alternative approaches to the problem.

Author response:
Most DM research studies, in the extant literature, have compared DM results against non metering baselines but, have avoided comparisons with other existing methods. The reason, most likely, is that differences in assumptions, airside modelling and problem formulations make the methods incompatible on the same data set. Moreover, the lack of data (often confidential) sharing and use of different data across studies, make it particularly difficult and unfair to compare solutions. However, we understand the importance of such a benchmarking effort, for knowledge creation, for the wider academic community. Thus, we have benchmarked (performance of) our DRL algorithm against Ali et al., 2022 and Mori 2017. Ali et al. 2022 is the state-of-the-art optimization method for DM with segregated runway operations and TS is the best planning algorithm for DM under uncertainty. Besides non metering baselines, these methods were chosen as they come closest to the idea of a numerical optimization problem, on simulated airside traffic data, where expected reward is optimized with respect to the policy’s parameters. Therefore, in principle, we believe, the existing comparisons are sufficient to show our method reliability and excellent performance.

Reviewer comment (Experiments):
Although the major contribution of this work appears to be the representation of MDP elements, and the authors do well in motivating their choices, it would be interesting to test how agnostic the proposed framework is to the choice of RL algorithm.

Author response:
Although not detailed in this paper due to space constraints, we conducted extensive experiments across various RL algorithms like DQN, A2C etc. We found that all RL algorithms learn and eventually converge. However, we found PPO to be the most stable and high-scoring across different traffic scenarios.

Reviewer comment (Experiments):
Unclear how method would transfer or scale to other airports

Author response:
The learnt DM policies cannot be directly applied to other airports due to different airside networks and operational constraints. However, If airport airside network information and historical surface movement data is provided, taxi routes and aircraft velocities at other airports can be simulated. Thereafter, our proposed DRL based DM framework can be easily adopted with little engineering effort which would involve clustering intersection nodes on the airside network into hotspots and applying real time trajectory prediction algorithm Tran et al. (2020) for generating a timed event graph. Thereafter, by tuning the reward coefficients ($\alpha_{1} - \alpha_{6}$), the DRL agent can be trained to perform DM, effectively balancing the airside congestion with runway utilisation, through interactions with the simulation environment.

Reviewer comment (Experiments):
How would the policy transfer to other airports with different layouts and traffic patterns?

Author response:
The learnt DM policies cannot be directly applied to other airports due to different airside networks and operational constraints. However, If airport airside network information and historical surface movement data is provided, taxi routes and aircraft velocities at other airports can be simulated. Thereafter, our proposed DRL based DM framework can be easily adopted with little engineering effort which would involve clustering intersection nodes on the airside network into hotspots and applying real time trajectory prediction algorithm Tran et al. (2020) for generating a timed event graph. Thereafter, by tuning the reward coefficients ($\alpha_{1} - \alpha_{6}$), the DRL agent can be trained to perform DM, effectively balancing the airside congestion with runway utilisation, through interactions with the simulation environment.

Reviewer comment (Experiments):
Could the event graph idea be used for other air traffic management tasks?

Author response:
Yes, the event graph idea can be potentially useful in air traffic Conflict detection and resolution (CD-R) tasks. Several studies on CD-R have employed a DRL-based approach to identify and resolve air conflicts. We can employ the timed event graph to predict aircraft trajectories, represented by a sequence of waypoints, within the DRL agent's state representation. Additionally, demand-capacity imbalance problems in air traffic management can be formulated using the event graph idea.

Reviewer comment (Experiments):
What are other ways to set the hyperparameters instead of manual tuning?

Author response:
Manual hyper-parameter tuning is one of the current limitations of reinforcement learning based techniques. Although not detailed in this paper due to space constraints, we conducted extensive experiments across various neural network architectures (2, 3, 4 layers), activation functions (tanh, relu, etc.), reward coefficients ($\alpha_{1} - \alpha_{6}$) etc., aiming to determine robust hyper-parameters suitable for different traffic scenarios.