Paper: NaviFormer: A Deep Reinforcement Learning Transformer-like Model to Holistically Solve the Navigation Problem (ICLR-2024)

Reviewer comment (Writing):
Although the proposed problem is a solid optimization problem, some unclear relation (navigation points of G, spraying points, and local/global maps, particularly with static obstacles) remains, which can be clarified by improving the writing.

Author response:
A1: A better explanation is offered as follows. The navigation points of $G$ are actually the same than the spraying points for the pesticide spraying task. The cultivation areas are segmented and divided into smaller sub-areas, represented by a single spraying point. Each spraying point must be visited by the UAV to cover a small part of each cultivation area. Once all the spraying points are visited, the cultivation area is completely sprayed. We have included an improved explanation about this topic at the second paragraph of section 5.

Regarding the local and global maps, the local maps show some small insight about the agent’s surroundings. Instead, the global maps contain a representation of the entire scenario. Again, the writing has been improved at the paragraph related to the ablation study, in section 5.

Reviewer comment (Experiments):
Insufficient experiment explanations, rather than giving some trivial figures: For example, in my opinion, some figures in Fig. 3 are not very important to convey the concept and idea of the paper (just following some basic idea of Transformer), but experiment information is much important as the author tries to tackle a new navigation problem. Fig. 6 should be updated for clarity without a 3D plot.

Author response:
A2: As a result of the reviewer’s suggestions, we have moved Figures 3.a and 3.b to Appendix A in order to gain more space to expand the results’ section. More specifically, we have included a comparison with a 2-step approach using the NaviFormer modules to certify the profit of jointly training the route planning and path planning modules. In addition, we have exposed some examples of the qualitative performance of NaviFormer with synthetic data. Moreover, we improved the writing of the results’ section to enhance the comprehension of the explanations and discussions provided.

Regarding Figure 6, it has been updated and now it does not contain any 3D plot.

Reviewer comment (Experiments):
Fixed training strategy in the main paper (no link or mention of different rewards or learning strategies).

Author response:
A3: We thank the reviewer for the suggestion, we will consider different training strategies, rewards, etc. in future research works. On the other hand, we employed a basic Actor-Critic strategy to face the deteriorating effect of sample variance during the learning process. The distribution of reward values of the nodes is fixed to a constant value of 1 because no node is preferred over the rest for our pesticide spraying application. However, multiple learning strategies and nodes could be utilized to train NaviFormer and achieve different performance results.

Reviewer comment (Experiments):
Compared with Fig. 1, the possible actions are restricted (only four angles). Can we discuss this point? For example, the learning difficulty when increasing the freedom of angles (e.g., 15 degrees each), the optimization problem, and results (e.g., more profitable routes with restricted times?). In the last sentence of the paper as a future work, could considering the continuous action space make the problem completely different (or some minor differences?).

Author response:
A4: Effectively, these are very pertinent issues to discuss. Increasing the freedom of angles would make the learning process more challenging because the number of solutions to the problem would increase in an exponential manner. In any case, it would be possible to learn the desired policy at expense of a lengthier training procedure and computational cost (for example, the computation of more local maps would be necessary). On the other hand, predicting continuous actions in a continuous action space would approximate better to the optimal solution. However, this type of solutions usually implies convergence issues during the training phase. This is because learning to find the best path to the next selected node using continuous actions requires much more precision, and the probability of predicting a “bad action” is much higher. As a consequence, we have specified in section 6 this limitation of our algorithm to encourage future research work.

Reviewer comment (Reproducibility):
Quick question: how did the authors design A* and D* for the problem? It is interesting if the authors give some details because, in my feeling, I’m not convinced that A* is time-consuming, as Table 2 says when the obstacle information is explicitly given. (Possibly, I misunderstood some settings).

Author response:
A7: A* is not time-consuming, as the reviewer has appreciated. However, it is important to note that the algorithms shown in Table 2 combine route planners (GA and OR-Tools) with path planners (A* and D*). And therefore, it can be stated that the majority of time consumed by the 2-step approaches that employ A*, is consumed by the route planners. In fact, the combination of OR-Tools and A* achieves the second smaller computation time, only surpassed by NaviFormer, which obtains a comparable computation speed.

Reviewer comment (Presentation):
Just a small question: what is an intuitive explanation or visualization of synthetic cases? (in real UAV datasets can be understood through Fig. 5).

Author response:
A8: We have included some synthetic examples showing the qualitative performance of the system in Figure 5.

Reviewer comment (Novelty):
The designed encoder that integrates the obstacle information into node embeddings using attention seems not new, which has already been proposed in [R].

Author response:
We would like to note that the definition of node used in [R] is different from our definition. In [R], nodes are predefined candidate positions that the agent can visit to follow a specific low-level trajectory that solves the path planning/finding problem. Instead, we solve the holistic navigation problem, which combines high-level routes and low-level paths, which are interrelated problems: the low-level paths affect the optimal strategy for the high-level paths (routes). Example: two close nodes that would seem reasonable to be visited can be separated by an obstacle making them in practice far away from each other. Therefore, in our nomenclature, low-level paths are a sequence of coordinates that an agent has to follow between consecutive nodes (but not a full route that satisfies a high-level purpose: maximize the number of visited nodes under restrictions of time). Comparatively, the high-level routes are a sequence of nodes that the agent has to visit, that is, regions to visit as in the orienteering problem or traveling salesman problem. For the pesticide spraying task, for example, each node is a spraying point (divisions of each cultivation to cover the entire area) where the agent has to perform a specific action (in this case, spray pesticide). And the trajectory to fly from one spraying node to the next one is a set of coordinates, that are not restricted to any fixed node position as in [R]. To better highlight the difference for each type of node definition, we will refer to them as low-level and high-level nodes.

Therefore, while it is true that the mentioned paper proposes a GNN encoder to integrate obstacle and low-level node information, our approach does not consider a predefined set of low-level nodes that the agent should follow (as [R] does). Therefore, our approach (which employs a Transformer encoder instead of a GNN) allows the agent to move freely around the scenario without any fixed low-level node position constraint. Besides, the mentioned GNN has to process a large set of low-level nodes to provide profitable paths, which seems to increase the computational time considerably, as the authors briefly comment in the results section, while our approach is focused on the computational efficiency of the estimated solutions. On the other hand, we just encode the information related to the high-level nodes and the obstacles and let the direction prediction layers to infer the low-level path. We can conclude that both methods have a common objective (integrating node and obstacle information) but follow different approaches to that end. They also address different problems: path planning [R] vs navigation planning (ours, which combines path and route planning).

Reviewer comment (Experiments):
The algorithm is only evaluated on two datasets, and the synthetic dataset lacks enough details. Besides, 640k training samples vs. 10k testing samplings is not a standardized data split, which may cause overfitting.

Author response:
In this regard, we have adopted the experimental setup of some well-known papers related to reinforcement learning applied to routing problems, such as [1] or [2], which use a similar data split, and therefore allows a better comparison of the results. In fact, the gap between training and testing samples is even larger in these cases: 1,280k vs 10k. In our humble opinion, in reinforcement learning scenarios, the most important issue to face is sample variance, which may prevent the system from learning properly and converging to good solutions. Therefore, very large training sets are necessary to guarantee good performance.

[1] W. Kool and H. van Hoof and M. Welling, Attention, Learn to Solve Routing Problems! ICLR 2019

[2] Irwan Bello, Hieu Pham, Quoc V. Le, Mohammad Norouzi, & Samy Bengio. (2017). Neural Combinatorial Optimization with Reinforcement Learning.

Reviewer comment (Writing):
It is not clear to me how the constraints described in Sec 3 are reflected in the neural architecture design. I hope the authors can give more clarification.

Author response:
Constraints described in section 3 essentially ensure that each node is not visited more than once, and that the agent returns to the end depot within the time limit. Since it is not possible to guess the distance between each pair of nodes in a fast and efficient manner, we cannot ensure that the end depot is reached on time. That is the reason to measure the success rate, which analyzes the number of times the agent bumps into an obstacle or does not reach the end depot on time. However, we can prevent the agent from revisiting nodes by including the mask $M$, detailed in equation 13. This mask will set the logit value corresponding to every visited node to $-\infty$, which is converted to a probability of 0 after the SoftMax operation. And that is the manner to include some of the constraints of section 3 into the neural architecture design. In summary, the neural network along with the training strategy are designed for satisfying the aforementioned restrictions, and depending on the specific restriction it is accomplished in a more implicit way or in a more explicit manner (masking).

Reviewer comment (Theory):
Is it necessary to predict both the future actions and directions? Does the predicted action imply the direction?

Author response:
Actually, the network predicts two types of actions: next node to visit and next direction to follow, since due to the obstacles, the direction to the next node could not be the same to that to the next waypoint. Therefore, we let the network to choose any visitable node and any direction. Later, we provide proper penalization/reward depending on the distance to the chosen node. Thus, the network learns to predict directions that minimize the distance to the chosen node.

Reviewer comment (Experiments):
Can NaviFormer deal with dynamic environment? i.e., the cases where the obstacles are moving.

Author response:
The scope of our paper only considers static obstacles. Therefore, as it can be observed in Figure 2, the graph embedding $h^{graph}$ is calculated one single time at the beginning, since the iterative calculations of $h^{graph}$ would not improve the performance in a static scenario. However, for a dynamic environment where obstacles are moving, iteratively calculating $h^{graph}$ would make sense, since each new embedding could indicate the presence of new obstacles or moving obstacles. This change in the network could imply a slight deterioration in computational cost, but it could allow solving the navigation problem with moving obstacles.

Reviewer comment (Experiments):
I see the joint training of the transformer and the network that predicts directions as the main contribution. What I am missing in that regard is an ablation of whether this actually works better than training both components in isolation. Right now it is hard to tell whether the routing patterns are really influenced meaningfully by the path-planning constraints. In that sense, I think the paper would be stronger if it considers a baseline where a routing-only transformer is pretrained like in prior art, and then a direction-prediction network is fit to its predicted goals post-mortem.

Author response:
A1: Following the reviewer suggestion, we have conducted an additional ablation test (see Table 1) comparing the performance of our proposed NaviFormer and a 2-step NaviFormer where the route planning submodule (base Transformer) is pretrained and used to fit the path planner (direction prediction layers). It can be observed that our proposal achieves a slight improvement in both success and node rates, which indicates that the route planner is lightly influenced by the path planner and viceversa. Besides, our 1-step approach is end-to-end trainable.

Reviewer comment (Experiments):
Related to the above, the paper would benefit from representative examples of what navigation runs look like, right now these are virtually missing (there is only one pesticide spraying example). This makes it hard to judge how well the routing and obstacle avoidance work together. Particularly because the considered environments are not that complex.

Author response:
A2: Figure 5 now includes several synthetic examples showing the qualitative performance of the system.

Reviewer comment (Experiments):
The overall idea to use a transformer for routing is directly carried over from prior art (e.g. [1]), one can easily tell by the used notation. Still, most of section 4 is about the transformer design. To count this as a contribution, I find the following points important: The ablation of the transformer (top 2 rows of table 1) is very important, to justify any changes in architecture.

Author response:
A3.1: We agree with the reviewer. For that reason, we have improved the explanations provided for the ablation study, which can be found at answer A3.3 for your convenience.

Reviewer comment (Novelty):
It appears that the traditional transformer baseline performs quite well in comparison, which makes me question whether the proposed layer structure (cross-attention, etc.) is really necessary.

Author response:
A3.2: As the reviewer has pointed out in question Q4, the proposed scenario is not as complex as other possible cases, such as apartment layouts or mazes. Therefore, the traditional encoder of the ablation study, which combines nodes and obstacles in a more simple and standard manner takes benefits from these scenarios with small number of obstacles to find good solutions to the problem. This fact can be corroborated by examining the success rate, which is higher for the NaviFormer encoder. This means that traditional encoders try to maximize the node rate even at the expense of risking the safe return to the end depot. In comparison, Naviformer is more cautious and tends to sacrifice a little bit the quantity of nodes to visit but ensuring the safe return of the asset. We believe that this policy is generally much more desirable, since collisions tend to be more catastrophic than smaller node rates in real-world applications.

Reviewer comment (Theory):
The direction-prediction net uses info only from the local neighborhood of the agent (based on figure 4). This makes the path-planning greedy, which may work reasonably in the considered experiments, but I doubt it will be optimal in something like apartment layouts or mazes. This should be highlighted as a limitation.

Author response:
A4: We agree with the reviewer that local maps may not be the best approach for more complex scenarios like apartment layouts or mazes, and it is a limitation that we have highlighted in the conclusions’ section to encourage future research.

Reviewer comment (Evaluation):
In terms of the runtime experiments: is the reported runtime of 3.5ms for one forward pass, or for predicting the whole set of goal waypoints + direction actions? It seems like the reported numbers for the traditional methods are for solving the whole problem.

Author response:
A5: The computation time reported for both the ablation study and the comparison with other state-of-the-art algorithms indicates the time required to solve the whole navigation problem to offer a fair comparison with other approaches. It is now indicated at the end of the ablation study, in section 5.

Reviewer comment (Presentation):
Some minor points: presentation-wise, while I appreciate the transparency in the network diagrams, I find these are better suited for an appendix, the information in them could be distilled. I also found the equations in sec. 3 somewhat too verbose, given that they are not used at all in the implementation (e.g. the transformer uses masking to implement most of the constraints).

Author response:
A6: Following the recommendation of the reviewer, we have moved Figure 3.a and 3.b to Appendix A.

Reviewer comment (Experiments):
When comparing to baselines, all approaches appear to be heuristic methods. It would have been useful to include a machine learning 2-step baselines as the improvement could simply be due to the different class of algorithm instead of the authors' assertion that NaviFormer beats baselines due to its tackling the routing and path planning problem jointly.

Author response:
We appreciate the reviewer suggestion. In that sense, we have conducted an additional experiment (see Table 1) comparing the performance of our proposed NaviFormer and a 2-step NaviFormer where the route planning submodule (base Transformer) is pretrained and used to fit the path planner (direction prediction layers). It can be observed that our proposal achieves a slight improvement in both success and node rates, which indicates that the route planner is lightly influenced by the path planner and viceversa. Besides, our 1-step approach is end-to-end trainable.

Reviewer comment (Evaluation):
How repeatable are the results over different seeds and validation set splits? It would have been useful to include a variance or confidence interval with the reported results.

Author response:
Following the suggestion of the reviewer, we have included the confidence interval with the results reported in Figure 6, Table 1, and Table 2. It can be observed that the intervals are very small, which suggests that results are potentially repeatable.

Reviewer comment (Writing):
At the beginning of section 4, $a_t^d$ can take the value $\frac{2\pi}{3}$. I assume this should be $\frac{3\pi}{2}$?

Author response:
The reviewer is right. It was a mistake, but now it is properly written in the paper.