Paper: Enhancing Airside Monitoring: Multi-view Approach for Accurate Aircraft Distance-To-Touchdown Estimation in Digital Towers (ICLR-2024)

Reviewer comment (Reproducibility):
The dataset is not available. Then it cannot be a part of contribution.

Author response:
The simulated dataset featuring multiple aircraft will be made publicly available upon request. However, the dataset obtained from real airport environments is considered highly confidential and cannot be shared.

Reviewer comment (Experiments):
The number of views in the dataset is only 2. The statement of "multi-view" is unsoundness. Author should increase the view number.

Author response:
Furthermore, the current choice of the number of views (e.g., 2) aligns with the current operational constraints of digital towers. However, Algorithm 1 is designed to accommodate arbitrary N (> 1):

As depicted in Algorithm 1, lines 7 and 8, during each iteration, each calibration network in the list of optimizing views connects to the same auxiliary network for training, aiming to minimize the prediction loss for distance to touchdown values. Moreover, as shown in lines 12-14 of Algorithm 1, views with loss values exceeding the minimum current loss of the calibration network by the defined threshold are selected. These views are then utilized for training in the next iteration. If all losses fall within a certain range, all calibration networks undergo training.

Reviewer comment (Novelty):
This manuscript sounds more like a technique report instead of a research paper. The proposed approach simply utilize an off-the-shelf detection model and an LSTM network to train a distance estimation model.

Author response:
Thank you for your constructive comments and feedback.

1. Calibration Network, Auxiliary Network, and Their Training:
The primary contribution of this work lies in proposing a framework that optimally utilizes multiple view camera streams in airport digital towers to estimate the distance to touchdown of landing aircraft. The approach is grounded in ensemble learning, where each view is processed independently, and their information is subsequently transformed into a shared space for ensemble prediction. The proposed framework is designed to provide flexible adaptation for different digital tower configurations. Depending on the available airport data and its corresponding challenges, the model for aircraft detection can be customized to maximize the system's performance. 

Besides, the calibration network plays a pivotal role in handling multiple cameras by converting input features from individual views into a shared feature space. Furthermore, an auxiliary network is shared and employed by all calibration networks to train their parameters.  During training, the calibration network is jointly trained with the auxiliary head, but it undergoes separate training sessions with other instances of the calibration network. Therefore after the training process, the weights of the calibration network become associated with the specific camera and its configuration, while the weight of the auxiliary head is general for every camera. The output of the calibration networks is considered as the shared feature space.

Furthermore, the current choice of the number of views (e.g., 2) aligns with the current operational constraints of digital towers. However, Algorithm 1 is designed to accommodate arbitrary N (> 1):

As depicted in Algorithm 1, lines 7 and 8, during each iteration, each calibration network in the list of optimizing views connects to the same auxiliary network for training, aiming to minimize the prediction loss for distance to touchdown values. Moreover, as shown in lines 12-14 of Algorithm 1, views with loss values exceeding the minimum current loss of the calibration network by the defined threshold are selected. These views are then utilized for training in the subsequent iteration until their loss value is within a said threshold. If all losses fall within a certain range, all calibration networks undergo training.

Reviewer comment (Experiments):
Majority of training and evaluation takes place in a setting that uses only two cameras. This is unsatisfactory given that the multi-view analysis is one of the central claims of this work.

Author response:
Furthermore, the current choice of the number of views (e.g., 2) aligns with the current operational constraints of digital towers. However, Algorithm 1 is designed to accommodate arbitrary N (> 1):

As depicted in Algorithm 1, lines 7 and 8, during each iteration, each calibration network in the list of optimizing views connects to the same auxiliary network for training, aiming to minimize the prediction loss for distance to touchdown values.

Moreover, as shown in lines 12-14 of Algorithm 1, views with loss values exceeding the minimum current loss of the calibration network by the defined threshold are selected. These views are then utilized for training in the next iteration. If all losses fall within a certain range, all calibration networks undergo training.

Reviewer comment (Presentation):
It is not immediately obvious how the architecture depicted in Figure 1 manages to integrate the information from multiple cameras. It seems that the "calibration network" is tasked with transforming the features captured by multiple cameras into a shared space where these can be reasoned with jointly. There is a need for much more discussion around this "calibration network" and how it helps integrate information from multiple cameras.

Author response:
And yes, the calibration network plays a pivotal role in handling multiple cameras by converting input features from individual views into a shared feature space. Furthermore, an auxiliary network is shared and employed by all calibration networks to train their parameters.

Reviewer comment (Reproducibility):
Part of the "inference" network contains an LSTM. It is not clear if LSTM is needed to deal with a single frame from multiple cameras or if LSTM is needed to process video feeds. It appears that temporal information may be helpful in regularizing the distance-to-touchdown estimates. It is unclear whether the system uses temporal information.

Author response:
In this study, Long Short-Term Memory (LSTM) is employed to combine inputs from multiple views for distance prediction, effectively handling variations in input size. The primary purpose of this model is to address situations where miss-detection occurs in one of the views. When YOLO fails to detect an aircraft or exhibits low confidence, the detected output is discarded, avoiding its use in subsequent prediction steps. This leads to variations in the input size during the inference step.

Reviewer comment (Writing):
The role of the auxiliary network is unclear. More importantly, it is not clear how it plays its role. The meaning of a "reversed network" is also unclear.

Author response:
The auxiliary network is a shallow network specifically crafted to extract information, such as distance-to-touchdown, from a shared feature space derived from individual views. A singular instance of this network is employed for all cameras. The rationale behind this design is based on the notion that if a shallow network consistently predicts the distance-to-touchdown from the shared feature space of different cameras, then that space must have effectively encoded the pertinent information.

Reviewer comment (Experiments):
The overall scheme seems rather ad hoc. YOLO is used as an object detector here — what if it fails to record a plane? What if planes are mis-labelled in multiple views? At a distance most planes look similar.

Author response:
In this study, Long Short-Term Memory (LSTM) is employed to combine inputs from multiple views for distance prediction, effectively handling variations in input size. The primary purpose of this model is to address situations where miss-detection occurs in one of the views. When YOLO fails to detect an aircraft or exhibits low confidence, the detected output is discarded, avoiding its use in subsequent prediction steps. This leads to variations in the input size during the inference step.

In cases of multiple aircraft in each view, even if the aircraft appearances are similar from a distance, their positions vary significantly based on their distance to touchdown. In these aircraft landing scenarios, all aircraft adhere to specific air traffic control procedures.

Reviewer comment (Experiments):
What happens if the N_views are set to more than 2 in Algorithm 1?

Author response:
Algorithm 1 is intended for use when N_views > 0, hence it should work as intended. In scenarios where multiple views exhibit loss values surpassing the threshold, the adaptive training algorithm will prioritize the view with the highest loss.

Reviewer comment (Experiments):
Why are max_epoches set to 225K in Algorithm 1? This seems an arbitrary number.

Author response:
Our observation indicates that, typically, the model achieves satisfactory performance without overfitting at 225k epochs. Nevertheless, it's essential to note that the optimal value of this epoch number may vary based on the specific characteristics of the data under consideration.

Reviewer comment (Evaluation):
Why does the system perform better in low-light conditions? Do we know why?

Author response:
We suspect that this phenomenon could be attributed to the contrast between the aircraft and its surroundings. The aircraft used in our experiments is a white Boeing 737, and this contrast may become more pronounced in darker skies, potentially contributing to the observed effects.

Reviewer comment (Reproducibility):
How does the system deal with the object association problem in multiple images?

Author response:
Currently, we employ a rule-based approach for associating aircraft across different views. We consider this method adequate, given the typically well-organized nature of air traffic.

Reviewer comment (Writing):
There is a lack of clarity in the Conclusions section: the second sentence refers to a stochastic number of input videos, but it is unclear what this actually means.

Author response:
This implies that the quantity of detections in video input may fluctuate from one timestep to another. Such variations can arise from either missed detections or instances where detection is excluded due to a confidence score falling below a specified threshold.

Reviewer comment (Writing):
The paper refers to "TensortRT". Is this intended to be "TensorRT"?

Author response:
We appreciate the clarification. We did intend to refer to TensorRT.

Reviewer comment (Experiments):
Can you provide some details of the calibration network? It may be useful to provide an ablative study on this, since it plays a central role in integrating information from multiple cameras.

Author response:
The calibration network is formulated to encode the distance-to-touchdown within a feature space derived from the aircraft bounding box, while simultaneously considering the properties and configuration of the camera. Consequently, the model acquires the ability to correlate the size and position of the bounding box in the frame with the aircraft distance. Recognizing that diverse camera configurations may influence this association, we employ N calibration networks for N different cameras, each specifically customized to the characteristics of the individual camera.

Reviewer comment (Writing):
What role does LSTM play? Is it used to combine information from multiple cameras, or does it integrate information over time to provide better distance-to-touchdown estimates?

Author response:
The LSTM plays a crucial role in this architecture by treating the N shared feature spaces from N camera views as a sequence with a length of N. In essence, the LSTM facilitates the integration of information across the camera views, allowing the model to generate a sensible prediction, albeit with reduced accuracy, in cases where missed detections occur.