Paper: Learning Coverage Paths in Unknown Environments with Reinforcement Learning (ICLR-2024)

Reviewer comment (Experiments):
My main concern is that the baselines may not be challenging enough. Intuitively, I agree on the focus on frontier-based baselines, but there should be a traveling salesman (TSP)-based baseline which aims to find a shortest path among the frontier points with a state-of-the-art TSP solver like Concorde, either in a discretized grid (converted into a graph) or in a probabilistic roadmap (PRM) graph. As mentioned in the survey below, TSP is a common technique applicable in coverage path problems.  
```
Tan, Chee Sheng, Rosmiwati Mohd-Mokhtar, and Mohd Rizal Arshad. "A comprehensive review of coverage path planning in robotics using classical and heuristic algorithms." *IEEE Access* 9 (2021): 119310-119342.
```

Author response:
__The baselines may not be challenging enough, a TSP-baseline should be included.__ We agree that a TSP-baseline would further increase the quality of the comparison. However, we believe that the methods under comparison provide sufficient insights about the performance of our method, as they span a wide range of approaches.

Reviewer comment (Experiments):
The RL-based baseline (Hu 2020) targets a multi-agent setting and is a simple feed-forward policy with no convolutional architecture, and it’s not clear if this is a fair comparison.

Author response:
__The RL-based baseline (Hu et al., 2020) targets a multi-agent setting, it is not clear if this is a fair comparison.__ As Hu et al. predict control signals using RL (although not end-to-end) we consider it the closest related method to our work, and thus we wanted to compare with it. Based on Voronoi partitioning, it can also be used for multi-agent exploration. However, for a fair comparison, we limited it to one agent as this is the focus of our paper. As Hu et al. predict the control signals based on the relative goal position from a previous stage in their framework, a simpler network architecture is sufficient, compared to predicting them based on the full map observation.

Reviewer comment (Experiments):
Overall, there is very little detail on how the baselines are implemented in this work; many of the baselines are originally proposed in multi-agent settings rather than single-agent settings. It’s difficult to judge how non-trivial the baselines are, and intuitively there should be a TSP-based or other combinatorial optimization-based baseline, given that CPP is a combinatorial problem. While the paper presents strong results relative to the given baselines, the baselines themselves have to be appropriately designed.

Author response:
__Unclear how the baselines are implemented.__ We used the implementations from their code repositories. We will provide links to these in the appendix to make it more clear.  
__Some baselines are originally proposed in multi-agent settings.__ Apart from Hu et al. and the potential field-based method (Yu et al., 2021), all other methods were originally proposed in a single-agent setting. Xu et al. (2022) later extended them to the multi-agent case. We believe that the absence of the multi-agent component in Yu et al. does not trivialize their method nor hinder single-agent performance, as they showcase their method also in single-agent exploration. This method still serves as a useful comparison in the single-agent case, as this is a common approach for single-robot path planning.

Reviewer comment (Writing):
I think the total variation contribution is slightly over-claimed, as the total variation is simply the perimeter of the explored area. While this shaping term is interesting and useful, using “total variation” to describe this reward makes the paper less clear and does not provide additional insights / intuition.

Author response:
__Total variation contribution is slightly overclaimed, it is simply the perimeter.__ The key is that TV can be computed locally and continuously, enabling e.g. variational image enhancement, infilling, and segmentation methods (a body of several hundred papers). The insight that these properties also enable its application in continuous RL is, to the best of our knowledge, novel.

Reviewer comment (Evaluation):
I see the collision penalty in the objective term. Does the agent collide at all with obstacles? If so, I would like to see the frequency of collision of the different approaches.

Author response:
__What is the collision frequency?__ For exploration, it varied between once every 100-1000 seconds, which is roughly once every 50-500 meters. For lawn mowing, it varied between once every 150-250 seconds, which is roughly once every 30-50 meters. However, the vast majority of collisions were near-parallel, and we didn’t see any head on collisions. The practical implications are very different between these cases. We will add this analysis to the appendix.

Reviewer comment (Novelty):
The research appears to have limited novelty, with a modest contribution.

Author response:
__The research has limited novelty, with a modest contribution.__ May we kindly ask the reviewer to be more specific, e.g. by giving a reference that would limit the novelty? As far as we know, previous work has not applied RL to learn continuous control signals end-to-end in CPP. In our interpretation, the strengths outlined by reviewers aapF and 5fHE tend to suggest that they agree. Learning control signals end-to-end in this way comes with unique challenges, which we address in the paper.

Reviewer comment (Novelty):
Most machine learning and classical planning methods can handle the investigated problem. It is hard to find new insights from this study.

Author response:
__Most machine learning and classical planning methods can handle the investigated problem.__ We agree that many classical and learning-based methods tackle a similar problem. However, none approach it end-to-end with RL as mentioned above. Compared to classical methods and multi-stage frameworks, this approach does not put any constraints on the path space, and allows the model to adapt to specific quirks in the agent dynamics in order to find a more efficient path.

Reviewer comment (Novelty):
In terms of multi-scale approaches, prior studies have delved into this issue, notably employing RL in autonomous driving as illustrated in reference [1].

Author response:
__In terms of multi-scale approaches, prior studies have delved into this issue.__ This is true. We are not suggesting that it is a new concept, but rather, that it is a necessary part in our solution to the challenge of applying RL to large-scale CPP. We will clarify this part in the paper.

Reviewer comment (Experiments):
How might the research tackle challenges related to sim-to-real transfer, especially considering the difficulties of implementing one-shot learning in real-world RL applications?

Author response:
__How might the research tackle sim-to-real transfer?__ Good question. As mentioned early on page 4, we believe that our approach can be used with an off-the-shelf SLAM method, removing the need for known pose. In this setting, the agent can learn and adapt to the noise characteristics during training. That our approach works well on noise-free data / data with simulated noise is a necessary condition for future real-world applications. The evaluation of our method on physical robot systems with realistic, noisy sensing will be subject of a future manuscript.

Reviewer comment (Reproducibility):
How does the agent architecture in this study differ from architectures used in methods like TRPO?

Author response:
__How does the agent architecture differ from architectures in methods like TRPO?__ The architectures used in TRPO are either feed-forward fully connected networks, similar to our MLP baseline, or convolutional networks for image observations, similar to the map feature branch in our CNN baseline. They are different in regards to our multimodal feature extractor and SGCNN architecture.

Reviewer comment (Novelty):
the paper provides as motivation with respect to classic methods "As these classical approaches cannot adapt to non-modeled environment dynamics, we turn to learning-based approaches", however, also the proposed method does not include any environment dynamics (the paper includes some examples in the introduction, such as damaged sensor or actuator). The paper currently lacks the corresponding gap from the classic methods and does not address the issue mentioned.

Author response:
__The proposed method does not include any environment dynamics.__ The ability to adapt to non-modeled environment dynamics was mainly used as a motivation for choosing an RL-based approach. Indeed, [1] shows that in grid environments, RL can be robust to dynamic environments and outperform classical methods. We missed to include this reference and will update the paper accordingly. Due to the findings in [1], we focused on the question whether an RL-based approach can learn patterns to choose locations that are promising in terms of new information in a continuous setting at all, before moving to dynamic environments.

[1] Saha et al. “Online area covering robot in unknown dynamic environments”, ICARA, 2021.

Reviewer comment (Experiments):
while the paper includes a number of different methods for comparison, there are some methods that are not discussed at all, but that however would be relevant to discuss, in particular some learning-based methods that are predicting the structure of the environments, such as  
Caley, Jeffrey A., Nicholas RJ Lawrance, and Geoffrey A. Hollinger. "Deep learning of structured environments for robot search." Autonomous Robots 43 (2019): 1695-1714.  
Shrestha, Rakesh, Fei-Peng Tian, Wei Feng, Ping Tan, and Richard Vaughan. "Learned map prediction for enhanced mobile robot exploration." In 2019 International Conference on Robotics and Automation (ICRA), pp. 1197-1204. IEEE, 2019.  
Arguably, compared to classic methods, a learning based approach can potentially learn some patterns to choose locations that are promising in terms of new information.

Author response:
__Some methods are not discussed at all.__ There are many works on CPP that share a connection to our work. However, to include all of them would not fit within 9 pages. We had to make a selection and choose the most related ones. We did not include Caley et al. (2019) as it tackles a slightly different problem, robot search (i.e. find the exit) instead of covering the area completely. Shrestha et al. (2019) presents and interesting approach for the same task and we thank the reviewer for this reference, we will include it in the related work.

Reviewer comment (Experiments):
there are parts that are not fully realistic, in particular the noise that would be present in the partial maps built by the robot, which has the effect of potentially guiding the robot towards areas that do not require exploration. It would be instead good to actually have the map built considering the noise. This can be achieved by using a realistic robot simulator with the perception/navigation stack already existing, e.g., in ROS.

Author response:
__The noise in the built maps are not fully realistic.__ As mentioned early on page 4, we believe that our approach can be used with an off-the-shelf SLAM method, removing the need for known pose. In this setting, the agent can learn and adapt to the noise characteristics during training. That our approach works well on noise-free data / data with simulated noise is a necessary condition for future real-world applications. The evaluation of our method on physical robot systems with realistic, noisy sensing will be subject of a future manuscript.

__It would be good to build the maps considering the noise using a realistic robot simulator, e.g. in ROS.__ We agree that this would be highly interesting, although ROS is not necessarily fully realistic as it does not have a full notion of continuous time. Evaluations on real robots would be preferable to any simulation.

Reviewer comment (Experiments):
it is also not clear why different methods are used for comparison in the omnidirectional and non-omnidirectional case, especially for the classic methods. Classic methods typically separate the high level decision and planning to the low-level control, so the same methods can be applied.

Author response:
__Why use different methods to compare omnidirectional/non-omnidirectional exploration?__ We simply compared the methods on the task setup described in their respective code implementations. We will provide links to the code repositories used in the appendix.

Reviewer comment (Evaluation):
to appreciate the difference between methods, it is important to include also the standard deviation, i.e., in Table I. It would have been good to include coverage over time also for the omnidirectional case, in the appendix.

Author response:
__The standard deviation should be included, e.g. in Table 1.__ We will do our best to do multiple runs for the standard deviation, although we can not promise that they will finish during the discussion period.

__It would be good to include coverage over time in the omnidirectional case in the appendix.__ Good suggestion. Although, the T90 and T99 metrics used in Explore-Bench convey information part of such a plot.

Reviewer comment (Experiments):
the multiscale component plays a role given the fixed w and h for the map, however, it is not clear the difference in performance between a multi-scale approach vs if those w and h would be high enough to capture the environment at enough fine resolution. It would be interesting to see such a difference. In addition, there might be environments where one scale is enough, while other complex environments require more than one scale. It would be interesting to discuss such a difference in different environments, thus hinting an adaptive scale, and how in practice an adaptive scale can be included, as the training will happen with specific scales.

Author response:
__How does a multi-scale map compare to a single map with high enough w and h?__ In large environments, using a single map is not feasible as the computational cost is O(n^2). For example, with our current setup with 4 maps that span an area of size 76.8x76.8 m2, the training step time is 40 ms. With a single map spanning only 19.2x19.2 m2 using the same pixel resolution as the finest scale, the training step time is 2500 ms. This would increase the total training time by roughly two orders of magnitude. Thus, a multi-scale map is actually necessary for large-scale environments. We will clarify this part in the paper.

__Could an adaptive scale be included?__ As you mention, the training needs to be done with specific scales. We believe the most practical way is to simply train with sufficiently many scales to cover the maximum size for a particular use case. If the model is deployed in a small area, the larger scales would not contain any frontier points in the far distance, but the agent can still cover the area by utilizing the smaller scales. Increasing the represented area by adding more scales is fairly cheap, as the computational cost is O(log(n)). We will add such a discussion to the appendix.

Reviewer comment (Evaluation):
what is the standard deviation of the results in Table I? in other words, are the differences statistically significant?

Author response:
__The standard deviation should be included, e.g. in Table 1.__ We will do our best to do multiple runs for the standard deviation, although we can not promise that they will finish during the discussion period.

Reviewer comment (Experiments):
I am not sure how practical the proposed approach is in real-world deployments - at the end of the day, the goal is to be able to deploy learned policies on real-systems. However, perfect observations for building global coverage/obstacle map and noise-free position/pose information are fairly strong assumptions that might limit practical utility. I do acknowledge the experiments with added Gaussian noise, but the scale of the noise isn’t completely clear and it seems like noise was only added to position information. I would have loved to see some real-world experiments or experiments on high-fidelity simulators with some discussion on inference speed/real-time performance.

Author response:
__Perfect observations and noise-free pose might limit practical utility.__ As mentioned early on page 4, we believe that our approach can be used with an off-the-shelf SLAM method, removing the need for known pose. In this setting, the agent can learn and adapt to the noise characteristics during training. That our approach works well on noise-free data / data with simulated noise (see below) is a necessary condition for future real-world applications. The evaluation of our method on physical robot systems with realistic, noisy sensing will be subject of a future manuscript.

__The scale of the noise isn’t completely clear.__ We measured the average position error for the SLAM method used in the CPP methods under comparison, and applied a similar noise to our method to get a fair comparison.

__Experiments on high-fidelity simulators.__ We agree that this would be highly interesting, although evaluations on real robots are preferable to any simulation.

Reviewer comment (Evaluation):
While I understand the work emphasizes on coverage path planning, I would have expected some discussion/experiments on the extent of collisions too - collisions matter in real-world problems, and unlike classical methods, the extent of collisions in a learned policy would at the end of the day depend on the relative weight of the collision-avoidance reward. It would be interesting to see the trade-off in the performance with more emphasis on collision avoidance when compared to classical approaches (to be clear, I am not recommending new experiments here - but it would be great if authors could discuss this aspect/share more details if they have the necessary information in the logs of existing runs).

Author response:
__What is the collision frequency?__ For exploration, it varied between once every 100-1000 seconds, which is roughly once every 50-500 meters. For lawn mowing, it varied between once every 150-250 seconds, which is roughly once every 30-50 meters. However, the vast majority of collisions were near-parallel, and we didn’t see any head on collisions. The practical implications are very different between these cases. We will add this analysis to the appendix.

Reviewer comment (Writing):
The word “environment dynamics” in the text is confusing - I understand what the authors wanted to convey but at the end of the day it’s agent dynamics.

Author response:
__The word “environment dynamics” is confusing, it should be “agent dynamics”.__ By “environment dynamics” we meant to convey aspects that do not only depend on the properties of the agent, such as wheel slip or moving obstacles.

Reviewer comment (Theory):
There are some issues with the notation in the POMDP definition. The state is Markovian by definition, and probability of transitioning to s_t should only depend on the previous state and action. If the problem formulation requires violating this, mention it explicitly.

Author response:
__In the POMDP definition, the probability of transitioning to s_t should only depend on the previous state and action.__ This is true, the state should not depend on all previous states and actions leading up to the new state (which we do not violate), we will update this.

Reviewer comment (Experiments):
Did you experiment with any other reward formulations besides coverage area and total variation?

Author response:
__Did you experiment with other reward formulations?__ Yes, we did. We tried a positive goal reward for reaching the goal coverage, and a negative truncation reward, but these gave no effect. We will include them in the appendix.

Reviewer comment (Experiments):
How does the performance scale to much larger environments? At some point the multi-scale representation may need higher resolution or more scales, right?

Author response:
__How does the performance scale to much larger environments? More maps or higher resolution is required, right?__ Yes, more maps or higher resolution would be needed for much larger environments. More maps would be most feasible, as the computational cost is O(log(n)). For example, our current setup with 4 maps spans an area of size 76.8x76.8 m2 with 4096 pixels. Increasing the number of maps to 6 would increase the computational cost by 50% while spanning more than 1x1 km2 in 6144 pixels.