Paper: Harmonized Learning with Concurrent Arbitration: A Brain-inspired Motion Planning Approach (ICLR-2024)

Reviewer comment (Experiments):
The author’s method requires a few critical hyperparameters which might be difficult to predefine in complex problems. The reliability threshold, being the most critical, appears to be extremely sensitive to not only the environment but also subtasks within an environment. Additionally, the perceived volatility parameter ti is rather arcane in how it is selected.

Author response:
The values for the reliability threshold $\alpha$ and the perceived volatility parameter $\tau$ lie within the range of $[0, 1]$. Theoretically, a larger threshold $\alpha$ leads to better results. However, setting a higher threshold will result in longer experiment runtime, with marginal or possibly no performance improvement. We adopt non-stringent thresholds, with $\alpha$ specifically set to 0.7. 

Given that different strategies are adaptable for dealing with distinct environments, we introduce the parameter $\tau$ to represent the preference for strategies in the inference buffer. In our experiments, we assume an equal likelihood for the selection of all strategies. Therefore, $\tau$ is set as 0.5. Besides, we provide details about the hyperparameters of the algorithm in Appendix C of the updated manuscript.

Reviewer comment (Writing):
The paper has several typos and requires additional proofreading to correct them, e.g., Section 4 line 2 “which can optimize the collection of training buffer and V (s; θ) and π(s ′ |s; θ) can be self-improving learned” or Section 4.3.1 last line of last paragraph “Thus, F(ω | Ut, I) does not require calculation and can be direct as a constant”, to name a few.

Author response:
We have improved the listed typos in the updated manuscript.

Reviewer comment (Evaluation):
There is no mention of the resources and time complexity the method requires versus the competition.

Author response:
We calculated Params for learning-based algorithms and demonstrated the planning runtime for all algorithms to find a successful path. 

 |Methods|Params| 
 |:----:|:----:|
|HLCA |0.16M |
|GNN+smoother |0.31M|
|GNN| 0.09M|
|NEXT|0.16M|

We demonstrated the total running time for 1000 test problems across all algorithms to find a successful path. A snapshot of the results is as follows. The complete table and discussion are updated in  Appendix A of the revised manuscript. To be more clearly represented in the figure, the results are normalized based on the running time of RRT*. HLCA, GNN+smoother, GNN, and NEXT are learning-based algorithms, others are non-learning algorithms.

 |$ $|Easy2|Hard2|Maze3|UR5|Kuka7|Kuka14|
 |:----:|:----:|:----:|:----:|:----:|:----:|:----:|
|HLCA |3.78 $\pm$ 0.04|5.86$\pm$ 0.11|3.64 $\pm$0.13|15.18$\pm$ 0.45|12.13$\pm$ 0.53 |10.78$\pm$ 0.42|
|GNN | 1.70$\pm$0.10|2.86$\pm$ 0.05|7.16 $\pm$ 0.42 |1.98$\pm$ 0.32|4.85$\pm$ 1.86|1.11$\pm$ 0.31|
|NEXT| 3.17$\pm$ 0.04|5.13$\pm$ 0.09|2.62$\pm$ 0.04 |23.49$\pm$ 0.06|18.57$\pm$ 1.00|15.89$\pm$ 0.13|
|BIT*|0.47$\pm$ 0.01|0.73$\pm$ 0.02|2.01$\pm$ 0.15|1.44$\pm$ 0.44|2.25$\pm$ 1.09|0.83 $\pm$ 0.43|
|LazySP|0.84$\pm$ 0.03|1.48$\pm$ 0.02|4.22$\pm$ 0.12 |2.27$\pm$ 0.17|1.93$\pm$ 0.01 |0.99$\pm$ 0.16|
| RRT*|1.00$\pm$ 0.01 |1.00$\pm$ 0.01|1.00$\pm$ 0.01 |1.00$\pm$ 0.00|1.00$\pm$ 0.03|1.00$\pm$ 0.01|

A common concern about learning-based algorithms is that their time cost due to the frequent calling of a large neural network model at the inference phase, such as NEXT. The running time is higher than non-learning algorithms because our algorithm is improved based on NEXT and runs multiple candidate strategies concurrently. Nevertheless, our algorithm reduces the running time compared with NEXT. Future work could focus on using pre-computation or caching, e.g., reusing past results to overcome some of this overhead.

Reviewer comment (Writing):
An algorithmic presentation of the algorithm is mandatory. The writing style made it rather hard to understand what is learned, when the strategies are rotated, etc.

Author response:
We have provided the pseudocodes of all our algorithms in Appendix E of the updated manuscript.

Reviewer comment (Writing):
It is confusing when the reliability checks happen for all strategies in I. Does a complete reliability assessment of all strategies happen when the current strategy k becomes unreliable or at each state expansion?

Author response:
We perform reliability inference for each planning problem, instead of before expanding states.

Reviewer comment (Reproducibility):
When a current strategy omega_k (learned) becomes inactive, and a new probing strategy is selected, how does this work? Is the new strategy omega_p initialized from omega_k and then further trained? Or are there several trainable strategies during initialization, and at each time a learnable strategy omega_k becomes unreliable, a new omega_p supplants it and is trained differently? The latter would mean that during the initial cycles more than one untrained strategy is selected and trained. Is that correct?

Author response:
(1) Brain model

In the human decision-making reasoning model, $\omega$ denotes a human behavioral strategy. When $\omega_{k}$ proves unreliable while other strategies in the inference buffer $\mathcal{I}$ remain reliable, the strategy with the highest reliability in $\mathcal{I}$ is selected. If none of the strategies in the inference buffer are unreliable, a new probe strategy $\omega_{p}$ is randomly developed from long-term memory. Notably, $\omega_{p}$ is not trained from the initial $\omega_{k}$ initialization.

(2) Algorithm

During the initialization phase, $\boldsymbol{\Omega}$ = {$\omega_{1},\omega_{2}, \omega_{3},\omega_{u}$} and $\mathcal{I}$ ={$\omega_{1},\omega_{2}, \omega_{u}$} were designed to imitate long-term memory and inference buffer, respectively, where $\omega_{1},\omega_{2}, \omega_{3}$ are existing strategies, the network structure of $\omega_{u}$ is existing. Similarly, If strategies in $\mathcal{I}$ are reliable, the strategy with the highest reliability is selected. In contrast, $\omega_{3}$ serves as a tentative probe strategy to guide agents in planning a path. This path is added to the training buffer as before. Moreover, if the strategies in $\mathcal{I}$ exhibit unreliability consecutively twice, $\omega_{3}$ replaces the strategy with the minimum reliability. Importantly, even in the scenario where $\omega_{u}$ is replaced, i.e., absent from the inference buffer, the samples collected persist in training $\omega_{u}$.

(3) Experiment

Observations from the experiments highlight that: two consecutive strategies in the inference buffer remain unreliable is a rare phenomenon. There are three reasons for this happen. (i) Our selected strategy demonstrates greater robustness. (ii) The threshold for reliability is not quite stringent. (iii) It is worth considering that the environments may not be quite complex.

For the third reason, we validated all methods by generating environments with larger dimensions and a higher density of obstacles. Experiment details can be found in Appendix D of the updated paper.

Reviewer comment (Presentation):
To me the derivation of the main algorithm was cluttered by discussion of (models of) brain function that I wasn't able to appreciate, more so as it was difficult for me to connect it to the description of the algorithm (which itself seemed to be presented rather loosely, relying on the figures and a sequence of paragraphs tending to different aspects, issues, and proposed solutions.) It would have helped to present the main logical steps in a pseudocode listing, with pointers to equations or section numbers that deal with finer issues.

Author response:
We have provided the pseudocodes of all our algorithms in Appendix E of the updated manuscript.

Reviewer comment (Experiments):
I also have concerns about the chosen benchmark leading the proposed algorithm to achieve 100% success rate. It is necessary to include a more challenging benchmark that specifically exposes failure cases, and compare to other baselines in that case too.

Author response:
Following the reviewers’ suggestion, we generated two groups of new test problems in the 2D environment, i.e., Dense2 and Large2. The environment settings are as follows: (i) Hard2: size, $15\times15$, obstacle density, no less than 0.46. (ii) Dense2: size, $15\times15$, obstacle density, no less than 0.60. (iii) Large2: size, $50\times50$, obstacle density, no less than 0.50. Maintaining consistent parameters across the board, we tested for all methods in new problems. The results are as follows and the complete table have been updated in the revised Appendix D. 

|Dense2|success rate|collision checks|path cost|running time|
|:----:|:----:|:----:|:----:|:----:|
|HLCA|0.99 $\pm$0.01|1959.04$\pm$22.88|2.56$\pm$0.02|12.18$\pm$0.55|
|GNN+smoother|0.96$\pm$0.01|2727.04$\pm$43.06|3.25$\pm$0.02|5.65$\pm$0.12|
|GNN|0.96$\pm$0.01|2304.78$\pm$41.51|3.96$\pm$0.02|5.45$\pm$0.12|
|NEXT|0.82$\pm$0.01|5032.77$\pm$50.21|2.38$\pm$0.02|15.02$\pm$1.89|
|BIT*|0.95$\pm$0.03 |2628.65$\pm$22.39|3.44$\pm$0.01|1.39$\pm$0.03|
|LazySP|0.99$\pm$0.01|2001.21$\pm$19.35|3.44$\pm$0.01|3.54$\pm$0.11|
|informted-RRT*|0.19$\pm$0.01|4911.51$\pm$ 20.36|1.80$\pm$0.04|1.07$\pm$0.02|
|RRT*|0.19$\pm$0.01|4929.20$\pm$30.44|1.79$\pm$0.04|1.00$\pm$0.00|

|Large2|success rate|collision checks|path cost|running time|
|:----:|:----:|:----:|:----:|:----:|
|HLCA|0.05$\pm$0.00|3256.82$\pm$10.25|2.52$\pm$0.04|68.53$\pm$1.26|
|GNN+smoother|0.02$\pm$0.00|5135.97$\pm$30.76|2.82$\pm$ 0.08|378.33$\pm$0.31|
|GNN|0.02$\pm$0.00| 5135.39$\pm$ 31.31|3.96$\pm$0.02|378.31$\pm$ 3.38|
|NEXT|0.00$\pm$0.00|/|/|85.80$\pm$1.28|
|BIT*|0.05$\pm$0.00|9843.90$\pm$129.08|3.22$\pm$0.47|9.80$\pm$0.22|
|LazySP|0.02$\pm$0.00|3162.30$\pm$15.08|3.44$\pm$0.01|3.54$\pm$0.11|
|informted-RRT*|0.69$\pm$0.01|2836.94$\pm$61.29|10.42$\pm$0.03|3.67$\pm$0.07|
|RRT*|0.00$\pm$0.00|/|/|1.00$\pm$0.03|

Compared to Hard2, operating in the Dense2 environment with a high obstacle density demands increased costs, collision checks, and runtime to reach the goal state. Our method performs best in terms of success rate and collision checks. While RRT* and informed-RRT* demonstrate a low path cost, their success rate lags behind other methods, possibly attributed to their efficacy in solving easy problems. NEXT exhibits lower path costs compared to other learning-based methods, but the collision checks are nearly twice as high.

For large environments, all algorithms demonstrate a notable reduction in success rate. Only a few instances where the distance between the start state and the goal state is not long succeed in finding a path. The majority of failures may be attributed to the limitation of the maximum number of samples, set as 1000. Particularly noteworthy is the superior performance of informed-RRT*, because informed sampling limits the sampling area and reduces the need for samples. For future work, we will try to set a higher maximum number of samples and introduce informed sampling to enhance the performance of our algorithm in large environments.

Reviewer comment (Evaluation):
It is necessary to include the computation/running-time of the motion planning algorithm itself, rather than just the number of collision-checks (also storage requirements for the memory buffers). Namely, as the proposed algorithm runs multiple candidate strategies in parallel, it is likely that its performance scales linearly with the number of active candidates. If pre-computation/caching, e.g., reusing past results is incorporated to overcome some of this overhead, then this deserves more discussion.

Author response:
We demonstrated the total running time for 1000 test problems across all algorithms to find a successful path. A snapshot of the results is as follows. The complete table is updated in  Appendix A of the revised manuscript. To be more clearly represented in the figure, the results are normalized based on the running time of RRT*. HLCA, GNN+smoother, GNN and NEXT are learning-based algorithms.

|$ $|Easy2|Hard2|Maze3|UR5|Kuka7|Kuka14|
|:----:|:----:|:----:|:----:|:----:|:----:|:----:|
|HLCA |3.78 $\pm$0.04|5.86$\pm$0.11|3.64$\pm$0.13|15.18$\pm$0.45|12.13$\pm$0.53|10.78$\pm$0.42|
|GNN|1.70$\pm$0.10|2.86$\pm$0.05|7.16 $\pm$0.42|1.98$\pm$0.32|4.85$\pm$1.86|1.11$\pm$ 0.31|
|NEXT|3.17$\pm$0.04|5.13$\pm$0.09|2.62$\pm$0.04|23.49$\pm$0.06|18.57$\pm$1.00|15.89$\pm$0.13|
|BIT*|0.47$\pm$0.01|0.73$\pm$0.02|2.01$\pm$0.15|1.44$\pm$0.44|2.25$\pm$1.09|0.83 $\pm$0.43|
|LazySP|0.84$\pm$0.03|1.48$\pm$0.02|4.22$\pm$0.12|2.27$\pm$0.17|1.93$\pm$0.01|0.99$\pm$0.16|
| RRT*|1.00$\pm$0.01|1.00$\pm$0.01|1.00$\pm$0.01|1.00$\pm$0.00|1.00$\pm$0.03|1.00$\pm$0.01|

A common concern about learning-based algorithms is that their time cost due to the frequent calling of a large neural network model at the inference phase, such as NEXT. The running time is higher than non-learning algorithms because our algorithm is improved based on NEXT and runs multiple candidate strategies concurrently. Nevertheless, our algorithm reduces the running time compared with NEXT. Future work could focus on using pre-computation or caching, e.g., reusing past results to overcome some of this overhead.

Reviewer comment (Evaluation):
Related to the above point, the maximum number of samples \( N_T \) ends up playing an important role in addressing the exploration-exploitation trade-off. I find that surprising as the proposed algorithm specifically aimed to offer new insights into this trade-off. Isn't it possible for the algorithm to produce running statistics indicating how much improvement can be expected if more samples are to be collected? Indeed, the ablation study with varying number of samples showed that performance reaches a plateau. Can the algorithm detect this? (e.g., in the course of arbitration as in S4.2.)

Author response:
Following the reviewers’ suggestion, we varied the maximum sampling number $N_T$ among {50, 100, 200, 500, 750, 1000, 2000, 4000, 6000}  on HLCA. The results are shown in Fig.9 of the updated Appendix B. From the observation of the results, HLCA reaches a plateau in all environments when $N_T$ is 500 in success rate and path cost. For failure cases, robots must generate more samples before successfully finding a collision-free path. This implies an increased demand for collision checks and the computation cost increases on both CPU and GPU.

Reviewer comment (Reproducibility):
The choice of the thresholds \( \beta \) and \( \rho \) seems to be absent.

Author response:
Theoretically, larger thresholds $\alpha$ and $\beta$ lead to better results. However, setting a higher threshold will result in longer experiment runtime, with marginal or possibly no performance improvement. During the experimental process, we try to strike a balance between efficiency and performance. $\alpha$ and $\beta$ are set as 0.7 and 0.9, respectively. Besides, we provide details about the hyperparameters of the algorithm in Appendix C of the updated manuscript.

Reviewer comment (Novelty):
The main concern is whether the brain models mentioned are the kind of science that can undergo significant revisions in the future, and whether the inclusion of such statements is essential to an article on RL algorithms in an AI/ML venue.

Author response:
Truly, the brain mechanism is the kind of science that can undergo significant revisions in the future. Within the last two years, there is still work in neuroscience that supports the brain mechanism of human decision-making reasoning [1, 2]. Following the reviewers’ suggestion, we make the relevant statement in S2 of the updated manuscript. "The brain mechanism about human decision-making reasoning upon which we draw in this study is confined to the outcomes that preceded our investigation."

Reviewer comment (Writing):
While it is beyond me to vet the contents of Section 2, I'm unable to take the assertions made about brain function at face value. First, I'd strongly recommend to preface this entire section by a clear statement of which articles this is drawn from. Then, it would help to qualify each assertion by specific references and statements of the form "current experimental evidence from neuroscience using brain scans/activation patterns suggests that", or "in experiments conducted on humans/animals on tasks involving", etc.

Author response:
We have clarified statements and references in S2. "The current experimental evidence from neuroscience using brain scans suggests that the concurrent reasoning mechanism in PFC includes two concurrent inference tracks [3]."

Reviewer comment (Writing):
In Section 3, the statement "... designed by imitating the inference buffer of the human brain" would be better referred to as "the XYZ model for inference in the human brain", rather than the "human brain" itself, with a reference for XYZ. Section 4.1 seems more in line with this request; this style should be followed consistently throughout.

Author response:
Following the reviewers’ suggestion, we changed it to "...is designed by imitating the inference buffer of the concurrent reasoning model in PFC [3]." in S3.

Reviewer comment (Writing):
In Section 4, the phrase "Motivated by the concurrent inference track in human PFC" should be addressed in the same way as the previous point.

Author response:
We changed it to "Motivated by concurrent inference tracks in the concurrent reasoning model  [3]" in S4.

Reviewer comment (Experiments):
In Section 4.3.2, I recommend replacing the number 200 with a hyperparameter, and explaining why this particular value was suitable for the experiments presented.

Author response:
Thanks and we have replaced 200 with the parameter $m$ in the updated manuscript and listed $m$ in Appendix C. The training size is also set to 200 in similar work [4, 5]. Chen et.al. [5] conducted further experiments to analyze the effect of the training size. There were 2000 training samples in their experiments. They find with 10% (200) problems of the original training set, the model is robust and obtains the best performance. Therefore, with the same number of training samples, we also set the training size to 200 in our experiments.

[4] Binghong Chen, Bo Dai, Qinjie Lin, Guo Ye, Han Liu, and Le Song. Learning to plan in high dimensions via neural exploration-exploitation trees. International Conference on Learning Representations, 2020.

[5] Chenning Yu and Sicun Gao. Reducing collision checking for sampling-based motion planning using graph neural networks. Advances in Neural Information Processing Systems, 34:4274–4289, 2021.

Reviewer comment (Presentation):
For Figures 11 & 12 in Section 5, it would help to include the metrics mentioned in the caption within the figure itself.

Author response:
We added the path cost and collision checks in the captions of Fig.11 and Fig.12.

Reviewer comment (Presentation):
In Appendix-A & B, the paragraph at the top appears redundant as it does not include meaningful conclusions from the results. It would be better to include a sentence or two about each table, beyond what's in the table captions. Perhaps that was all deferred to Appendix-B. If that's the case, it would help to communicate this structure, though I'd recommend to keep each paragraph next to the relevant table. (Prefer to have the conclusions closer to the supporting evidence.)

Author response:
Following the reviewers’ suggestion, we have merged the former Appendix A and B into Appendix A in the updated manuscript. The paragraphs in the former Appendix B have been relocated beside the corresponding table.

Reviewer comment (Experiments):
In Appendix-A & B, Table 1 shows that only few cells have non-zero stdev. A comment about that would be helpful, and I wonder if this indicates that experiment configurations need to be revised to include more challenging cases over all environments. Specifically, it would help to supplement this table with a new set of experiments (and table) designed to show HLCA failure cases, e.g., average success rate near 80%, and show how far other baselines regress for the same test cases.

Author response:
In addition to the consideration that the environment may not be quiet challenging, only a few cells have non-zero stdev is also linked to the calculation of the success rate. The success rate represents the proportion of successful paths found over 1000 test problems. Although the stdev in some cells exhibits 0.00, the success paths differ by more than 40 in different randomized seed scenarios. Following the reviewers’ suggestion, we generated two groups of more challenging benchmarks. Details can be found in reply 4.

Reviewer comment (Experiments):
In Appendix-A & B, Table 3 shows the gap in HLCA is highest for UR5. I wonder if this indicates different priors/hyperparameters are needed for this environment. It would help to include a comment about how this gap may be reduced by specialization of the proposed approach.

Author response:
Hyperparameters are kept consistent across all methods and all environments in this experiment. Unlike other robotic arm environments, obstacles are mainly concentrated around the robotic arm in UR5. There are two reasons why the gap in HLCA is highest for UR5. (i) Tree-based sampling methods such as RRT* have lower success rates due to sampling from the neighborhoods. For the failure cases, the HLCA needs to spend more path cost to be successful. (ii) We prioritize the reduction of collision checks and make some sacrifices in terms of path cost. 

Hence, it is reasonable that the gap can be narrowed through these two ways. First, the sampling scale can be extended when expanding a new state. Second, the path cost can be given a higher weight in the process of making reliability inferences.

Reviewer comment (Presentation):
In Appendix-C, please indicate that this entire section is focused on the HCIL method, and revise figure captions to mention this as was done for Fig.9. Looking into this again, it's not immediately clear how the first paragraph+Fig.8 differs from the second paragraph+Fig.9.

Author response:
Sorry for the lack of clarity. The title of Fig.8 has been revised in the updated manuscript, accompanied by the inclusion of a paragraph elucidating the purpose of this section. "We investigate the inference of the maximum sampling number $N_T$ on CAM and HLCA, respectively."

Reviewer comment (Writing):
In Section 3.1, there is an unmatched parenthesis in the definition of \( U_i \).

Author response:
We have improved all the listed typos in the updated manuscript.

Reviewer comment (Writing):
In Section 4.3.1, the phrase "be direct as a constant" should perhaps be "be directly used as?"

Author response:
We have improved all the listed typos in the updated manuscript.

Reviewer comment (Writing):
In Section 4.3.2, many occurrences of "cycle" are better replaced with "iteration" or "epoch".

Author response:
We have improved all the listed typos in the updated manuscript.

Reviewer comment (Reproducibility):
The authors do not provide sufficient details about the parameters of the algorithm to compare with, which impairs the credibility of the experimental results. In addition, the runtime for success rates must be reported as well.

Author response:
We have provided details about the parameters of the algorithm in Appendix C of the revised manuscript. These parameters  include learning rates, epochs, reliability thresholds $\alpha$, $\beta$, and so on.

2. Total running time

We demonstrated the total running time for 1000 test problems across all algorithms to find a successful path. A snapshot of the results is as follows. The complete table is updated in  Appendix A of the revised manuscript. To be more clearly represented in the figure, the results are normalized based on the running time of RRT*. HLCA, GNN+smoother, GNN, and NEXT are learning-based algorithms, others are non-learning algorithms.

|$ $|Easy2|Hard2|Maze3|UR5|Kuka7|Kuka14|
|:----:|:----:|:----:|:----:|:----:|:----:|:----:|
|HLCA |3.78 $\pm$0.04|5.86$\pm$0.11|3.64$\pm$0.13|15.18$\pm$0.45|12.13$\pm$0.53|10.78$\pm$0.42|
|GNN|1.70$\pm$0.10|2.86$\pm$0.05|7.16 $\pm$0.42|1.98$\pm$0.32|4.85$\pm$1.86|1.11$\pm$ 0.31|
|NEXT|3.17$\pm$0.04|5.13$\pm$0.09|2.62$\pm$0.04|23.49$\pm$0.06|18.57$\pm$1.00|15.89$\pm$0.13|
|BIT*|0.47$\pm$0.01|0.73$\pm$0.02|2.01$\pm$0.15|1.44$\pm$0.44|2.25$\pm$1.09|0.83 $\pm$0.43|
|LazySP|0.84$\pm$0.03|1.48$\pm$0.02|4.22$\pm$0.12|2.27$\pm$0.17|1.93$\pm$0.01|0.99$\pm$0.16|
| RRT*|1.00$\pm$0.01|1.00$\pm$0.01|1.00$\pm$0.01|1.00$\pm$0.00|1.00$\pm$0.03|1.00$\pm$0.01|

A common concern about learning-based algorithms is that their time cost due to the frequent calling of a large neural network model at the inference phase, such as NEXT. The running time is higher than non-learning algorithms because our algorithm is improved based on NEXT and runs multiple candidate strategies concurrently. Nevertheless, our algorithm reduces the running time compared with NEXT. Future work could focus on using pre-computation or caching, e.g., reusing past results to overcome some of this overhead.

Reviewer comment (Experiments):
There are other learning-based algorithms to compare with. The reviewer would like to see the comparison results against those learning methods: Zhang, Ruipeng, et al. "Learning-based Motion Planning in Dynamic Environments Using GNNs and Temporal Encoding." Advances in Neural Information Processing Systems 35 (2022): 30003-30015.

Author response:
Following the reviewers’ suggestion, we have implemented GNN-TE[1] as another baseline method to compare with. A snapshot of the results is as follows, and we will update the manuscript with the complete tables. 

Average collision checks.

 |$ $|Easy2|Hard2|Kuka7| Kuka13|Kuka14|
 |:----:|:----:|:----:|:----:|:----:|:----:|
|HLCA |129.05$\pm$ 3.04|342.35$\pm$ 13.90|479.86$\pm$ 63.64|500.43$\pm$ 269.34|508.03$\pm$52.56|
|GNN-TE|412.66$\pm$ 8.16|659.90$\pm$ 13.52|537.08$\pm$ 61.64|498.25$\pm$56.89|497.20$\pm$ 89.22|
|GNN | 416.54$\pm$ 9.23|720.47$\pm$ 7.46 |633.79$\pm$ 60.39|522.87$\pm$64.25|564.92$\pm$ 100.08|
|NEXT|272.28$\pm$ 14.81|1135.6$\pm$ 37.7 |2516.75 $\pm$ 45.73|4870.57$\pm$ 129.12|4557.94 $\pm$ 71.22|

Average path cost.

 |$ $|Easy2|Hard2|Kuka7| Kuka13|Kuka14|
 |:----:|:----:|:----:|:----:|:----:|:----:|
|HLCA |1.08$\pm$ 0.00|2.01$\pm$ 0.00|6.30$\pm$ 0.03|9.67$\pm$ 0.04|9.78$\pm$ 0.09|
|GNN-TE|1.51$\pm$ 0.00|2.80$\pm$ 0.01|8.96$\pm$ 0.11|16.88$\pm$ 0.12|16.69$\pm$ 0.22|
|GNN | 1.50$\pm$ 0.00|2.89$\pm$ 0.01|9.15$\pm$ 0.07|17.19$\pm$ 0.23|16.83$\pm$ 0.26|
|NEXT|1.06$\pm$ 0.00|2.04$\pm$ 0.00|7.90$\pm$ 0.05|10.25$\pm$ 0.11|10.73$\pm$ 0.09|

After a careful reading, it is noted that the GNN-TE is particularly designed for dynamic environments. With full knowledge of the obstacles, GNN-TE leverages graph neural networks and temporal encoding framework to achieve motion planning problems in dynamic environments. However, our work mainly concentrates on static environments, in which the GNN-TE obtains approximate performance of the GNN. It is expected that GNN-TE will perform better in dynamic environments, which would be our future study topic.

[1] Zhang, Ruipeng, et al. "Learning-based Motion Planning in Dynamic Environments Using GNNs and Temporal Encoding." Advances in Neural Information Processing Systems 35 (2022): 30003-30015.

Reviewer comment (Writing):
Minor: Section 3.1 miss left parenthesis.

Author response:
We have improved the typo mentioned in minor comments in the revised manuscript.

Reviewer comment (Evaluation):
In Figure 4(a), I am curious about why success rates all drop down to 0.4 for Ur5, which are supposed to be quite easy for sampling algorithms such as RRT*.

Author response:
There are two reasons why this happens. Unlike other robotic arm environments, obstacles are mainly concentrated around the robotic arm in UR5. On the one hand, RRT* samples a state from free space, and determines a new state to connect to through the steer function. The steer function only generates states that are close to the search tree within neighboring areas. Therefore, in environments with high obstacle density, tree-based sampling methods such as RRT* struggle to expand new states. On the other hand, the maximum sample number is limited to 1000 in our experiments. Under this condition, RRT* reaches the maximum budget of the sampling nodes and is forced to be terminated without finding a solution. Consequently, it directly results in a substantial decrease in the success rate of RRT*.

Reviewer comment (Experiments):
The benchmarked tasks are all short-horizon problems. These tasks can be quite easy for the algorithms to compare with. The reviewer would like to see to larger 2D maze map or more obstacle-rich arm motion planning problems.

Author response:
Following the reviewers’ suggestion, we generated two groups of new test problems in the 2D environment, i.e., Dense2 and Large2. The environment settings are as follows: (i) Hard2: size, $15\times 15$, obstacle density, no less than 0.46. (ii) Dense2: size, $15\times 15$, obstacle density, no less than 0.60. (iii) Large2: size, $50\times 50$, obstacle density, no less than 0.50. Maintaining consistent parameters across the board, we tested for all methods in new problems. The results are as follows and the complete table have been updated in the revised Appendix D. 

|Dense2|success rate|collision checks|path cost|running time|
|:----:|:----:|:----:|:----:| :----:|
|HLCA|0.99 $\pm$0.01|1959.04$\pm$22.88|2.56$\pm$0.02|12.18$\pm$0.55|
|GNN+smoother|0.96$\pm$0.01|2727.04$\pm$43.06|3.25$\pm$0.02|5.65$\pm$0.12|
|GNN|0.96$\pm$0.01|2304.78$\pm$41.51|3.96$\pm$0.02|5.45$\pm$0.12|
|NEXT|0.82$\pm$0.01|5032.77$\pm$50.21|2.38$\pm$0.02|15.02$\pm$1.89|
|BIT*|0.95$\pm$0.03 |2628.65$\pm$22.39|3.44$\pm$0.01|1.39$\pm$0.03|
|LazySP|0.99$\pm$0.01|2001.21$\pm$19.35|3.44$\pm$0.01|3.54$\pm$0.11|
|informted-RRT*|0.19$\pm$0.01|4911.51$\pm$ 20.36|1.80$\pm$0.04|1.07$\pm$0.02|
|RRT*|0.19$\pm$0.01|4929.20$\pm$30.44|1.79$\pm$0.04|1.00$\pm$0.00|

|Large2|success rate|collision checks|path cost|running time|
|:----:|:----:|:----:|:----:|:----:|
|HLCA|0.05$\pm$0.00|3256.82$\pm$10.25|2.52$\pm$0.04|68.53$\pm$1.26|
|GNN+smoother|0.02$\pm$0.00|5135.97$\pm$30.76|2.82$\pm$ 0.08|378.33$\pm$0.31|
|GNN|0.02$\pm$0.00| 5135.39$\pm$ 31.31|3.96$\pm$0.02|378.31$\pm$ 3.38|
|NEXT|0.00$\pm$0.00|/|/|85.80$\pm$1.28|
|BIT*|0.05$\pm$0.00|9843.90$\pm$129.08|3.22$\pm$0.47|9.80$\pm$0.22|
|LazySP|0.02$\pm$0.00|3162.30$\pm$15.08|3.44$\pm$0.01|3.54$\pm$0.11|
|informted-RRT*|0.69$\pm$0.01|2836.94$\pm$61.29|10.42$\pm$0.03|3.67$\pm$0.07|
|RRT*|0.00$\pm$0.00|/|/|1.00$\pm$0.03|

Compared to Hard2, operating in the Dense2 environment with a high obstacle density demands increased costs, collision checks, and runtime to reach the goal state. Our method performs best in terms of success rate and collision checks. While RRT* and informed-RRT* demonstrate a low path cost, their success rate lags behind other methods, possibly attributed to their efficacy in solving easy problems. NEXT exhibits lower path costs compared to other learning-based methods, but the collision checks are nearly twice as high.