Paper: Grid Cell-Inspired Fragmentation and Recall for Efficient Map Building (ICLR-2024)

Reviewer comment (Evaluation):
the improvement over the alternative methods seems to shrink in the more real-world-like applications. Can the authors comment on or provide an interpretation of this?

Author response:
In robot simulations, there is significant noise from the LIDAR sensor, path planning, actuation, and localization of the current position. These factors lead to increased memory usage in both the Frontier and FARMap methods. Consequently, the performance gap between these two methods is smaller than in the proposed environments. Additionally, we measured the memory usage of FARMap based on its peak usage during navigation. While the maximum usage of FARMap is similar to its counterpart in Environment 1 and in American settings, the overall usage is consistently lower. For instance, in AWS, FARMap explores 49% more of the environment while using only 37% more memory.

In robot simulation, there are many noise from LIDAR sensor, path planning, actuating, and localizing current position. They lead to use more memory on both Frontier and FARMap. Therefore, the gap between two methods is smaller than the proposed environments. On the other hand, we measure the memory usage of FARMap as the largest memory usage in navigation. Although the maximum usage of FARMap is similar to its counterpart in Environment 1, and American, the overall usage is smaller. In AWS, FARMap explores 49% more environment while it uses 37% more memory usage.

Reviewer comment (Experiments):
Can the authors justify why the Frontier method by Yamauchi (1997) would be the most appropriate benchmark to have? It would be nice if it’s possible to include some other more recent methods.

Author response:
We selected the Frontier method (Yamauchi, 1997) as our baseline because it is a fundamental technique widely used in contemporary methods (Kulkarni et al., 2022) and is often employed as a benchmark (Chaplot et al., 2020). Its widespread adoption positions the Frontier method as an ideal candidate for integration with FARMap. This integration facilitates the possibility of employing various methods for local subgoal generation, thereby enhancing the versatility and applicability of FARMap.

Reviewer comment (Writing):
In addition, I am not sure if “grid cell-inspired” in the title is entirely appropriate given that the authors do not use grid cells in their model and the idea of representing space in segments seems to hold more generally for the hippocampal-parahipponcampal representation in rodents.

Author response:
As the reviewer pointed out, the concept of remapping is indeed associated with place cells in the hippocampus, not exclusively with grid cells (as noted in the second paragraph of our Introduction). However, our approach employs surprisal changes as a criterion for fragmentation, which correlates with the locations of actual grid cell remapping, as shown in existing studies (Klukas et al., 2021). Therefore, we titled our paper “grid cell-inspired”. We will change the paper title reflecting on your comment if the paper is accepted.

Reviewer comment (Novelty):
while it is interesting to see the grounding of the proposed method in neuroscience, some of the general ideas are already present in other methods for exploration, in particular, reasoning topologically is captured by methods that use the generalized Voronoi graph or semantic maps to guide the exploration, and the long-term storage through pose graphs in SLAM, where loop closure is applied (discussed in graph-based slam appendix section), or curiosity-driven exploration. The paper should discuss the proposed method with respect to such methods.

Author response:
We appreciate the reviewer's acknowledgment of our discussion on graph-based SLAM in Appendix A. The topological graph in FARMap and the Generalized Voronoi Graph (GVG)-based approaches are designed for different objectives. GVG-based methods primarily aim for rapid path planning through pre-computed paths and loop closure. In contrast, FARMap's topological graph is designed to surmount the limitations of local maps that cannot refer to other submaps. However, this does not mean that both methods cannot be used together. Rather, there is a potential synergy to use both as we mentioned in Appendix A.1.

Regarding curiosity-driven exploration [1,2], these methods employ future prediction error or surprisal as intrinsic rewards to address environments with sparse rewards. Conversely, FARMap utilizes surprisal for segmenting space in accordance with areas where grid cell remapping occurs.

[1] Pathak, Deepak, et al. "Curiosity-driven exploration by self-supervised prediction." *ICML*, 2017.

[2] Burda, Yuri, et al. "Exploration by random network distillation." ICLR, 2019.

Reviewer comment (Evaluation):
the gain in memory appears to be a major component of the proposed method, however, overall, the trend seems to be fairly close to the frontier-based approach and somewhat surprising given the use of local maps. In fact, for the realistic experiments, in AWS office, memory appears better for Frontier. The size of each local map might depend on the complexity of the environment, but it is worth discussing what affects the determination of the local map in practice.

Author response:
We acknowledge that in the AWS office scenario, FARMap demonstrates higher memory usage compared to Frontier, contrary to the results in our proposed environments where FARMap and Frontier have a significant gap in memory usage but achieve better map coverage. However, it's important to consider that coverage directly impacts memory usage. In AWS, FARMap explores 49% more of the environment while using only 37% more memory, showcasing its efficiency in broader exploration.

Reviewer comment (Writing):
The map representation is not clearly explained. The information provided appears to be contradicting itself: the C-th channel is said to contain confidence information, but it is unclear over what; additionally, the C-th channel in the observation contains visibility information. Later, there is talk of occupancy and colors. The actual representation used by the maps is never concretely explained.

Author response:
We apologize for the confusion regarding the local map and surprisal. The local predictive spatial map is a $C+1 \times H \times W$ dimensional tensor. In each coordinate (h, w), it represents C+1 dimensional vectors, where the first C dimensions (0 to C-1) denote color and the last dimension denotes confidence. If the observation is RGB as in the proposed environments, C is 3. Similarly, C is 1 if the observation is in grayscale. On the other hand, if the observation comes from an occupancy sensor (i.e., LIDAR in Section 5.2 and Neural SLAM module in Section 5.3), C is 0, which means that there is no color. We will clarify this in the revision.  

The value of the C-th dimension represents the confidence of each cell observed during exploration. It is gradually decaying like a biological agent’s working memory as we mentioned in Section 3.1. We kindly remind you that the primary goal of this paper is to apply neuroscience theory and foundations to SLAM.  

Please refer to the attached codebase (lib/memories/memory.py) for the detailed actual implementation. If you have further questions, please let us know. We will be happy to answer them.

Reviewer comment (Theory):
The surprisal mechanic lacks clarity. It mentions uncertainty estimation yet provides the prediction error as an example, raising the question of how an error can be used as uncertainty estimation. Equation 2 is highly confusing as M and o are matrices with different dimensionality yet are multiplied together, and the multiplied quantities represent different properties, adding to the confusion. Without a clear explanation of what local maps look like and how they are formed, the entire surprisal aspect is challenging to evaluate.

Author response:
The C-th channel of the local predictive map, $\mathbf{M}^\text{cur}_{t-1},C \in \mathbb{R}^{H\times W}$, where H and W represent the height and width of the map, contains the confidence level of each cell. If a cell has been observed before, it holds a nonzero value; otherwise, its value is 0. On the other hand, the C-th channel of the spatially transformed observation at time $t$, denoted as $o’_t,C \in \mathbb{R}^{H\times W}$ (refer to Figure 3 for a visual representation), indicates whether a cell is currently visible (1) or not (0).  

Therefore, $\mathbf{M}^\text{cur}_{t-1}, C \cdot o’_t,C$ represents the summation of the confidence of each visible cell in the local predictive map before updating the current (time $t$) observation. Similarly, $|| o’_t,C ||_1$ represents the number of visible cells in the current observation. Hence, Eq. (2) denotes the average confidence of a visible cell from the current observation.  

Surprisal is usually defined as future prediction error or distribution difference in many research areas such as psychology [1], reinforcement learning [2], and neuroscience [3]. However, surprisal can also be interpreted as uncertainty (predicted variance), as demonstrated by Kendal et al. [4]. Therefore, we refer to it as ‘uncertainty estimation’ in our work. We will revise the terminology for clarity in our revised manuscript.  

[1] Modirshanechi, Alireza, Johanni Brea, and Wulfram Gerstner. "A taxonomy of surprise definitions." *Journal of Mathematical Psychology* 110 (2022): 102712.  

[2] Achiam, Joshua, and Shankar Sastry. "Surprise-based intrinsic motivation for deep reinforcement learning." *arXiv preprint arXiv:1703.01732* (2017).  

[3] Sinclair, Alyssa H., et al. "Prediction errors disrupt hippocampal representations and update episodic memories." *PNAS* 118.51 (2021): e2117625118.  

[4] Kendall, Alex, Yarin Gal, and Roberto Cipolla. "Multi-task learning using uncertainty to weigh losses for scene geometry and semantics." *CVPR*. 2018.

Reviewer comment (Reproducibility):
The recall aspect, which is paramount to reusing existing local maps efficiently, lacks any information regarding how it works. It is unclear whether the system assumes perfect localization and can just use the submap graph, or if there is a place recognition system that reidentifies these local submaps.

Author response:
We do not assume perfect localization. Results from Robot simulations using a LIDAR sensor (Section 5.2) and Habitat simulations with the Neural SLAM module, which predicts 2D observations from RGB egocentric observations (Section 5.3), demonstrate that our method works well in noisy, continuous environments without assuming perfect localization.

Reviewer comment (Evaluation):
The metrics used in the experiments are unclear and hard to interpret. For example, Table 2 shows memory usage with a unit of (k), which is not explained. The statistical information provided switches between standard deviation and quantile instead of sticking to one (likely quantile).

Author response:
In Table 2, 'k' is used to denote 1000, representing the actual discovered size of the environment (in cell units with the resolution mentioned in Appendix D.2). Regarding resolution, we use the default resolution of turtlebot, 0.05 m x 0.05 m. We will modify the notation to (x1000) in future revisions and convert these values into square meters (m^2) for clarity.  

Thank you for your comment. We employed a 95% confidence interval from bootstrapping for the proposed environments, as the results are aggregated from various maps not from the same map (distribution), making standard deviation inappropriate. Conversely, in the robot simulation in ROS, we can calculate a 95% confidence interval based on standard deviation as it was run multiple times in the same environments. We will adjust the notation of the 95% confidence interval in Table 2 accordingly.

Reviewer comment (Experiments):
From the description of the baseline method, it is unclear whether it also uses submaps. The results seem to imply so, as otherwise, the relative memory plots should show a value of 1 based on the reviewer’s understanding.

Author response:
The baseline method does not utilize a submap-based approach for identifying subgoals and planning. However, as mentioned in Appendix A.1, FARMap can be combined with any planners that employ graph- or submap-based approaches for path planning. For instance, GBPlanner uses a frontier method to select a subgoal, while employing a graph-based approach for rapid path planning. It can be integrated with FARMap by replacing the Planner component (such as Dijkstra’s algorithm or RRT) and the subgoal identification mechanism (Frontier) with those from GBPlanner.