Paper:
Visual Memory for Robust Path Following
Humans routinely retrace paths in a novel environment both forwards and backwards despite uncertainty in their motion. This paper presents an approach for doing so. Given a demonstration of a path, a first network generates a path abstraction. Equipped with this abstraction, a second network observes the world and decides how to act to retrace the path under noisy actuation and a changing environment. The two networks are optimized end-to-end at training time. We evaluate the method in two realistic simulators, performing path following and homing under actuation noise and environmental changes. Our experiments show that our approach outperforms classical approaches and other learning based baselines.

Review:
This paper considers the problem of retrace the trajectory from noisy visual observations. The proposed approach firstly takes a sequence of images of the path and encode as sequential memories. In the online setting, a neural network is able to attend to the past memory and take actions accordingly. The algorithm has been verified over two interesting 3D indoor scene dataset.    Pros:    * The paper is very well-written. Introduction is clearly motivated, novelty and relationship with previous works have been discussed in a very modest tone, approach is very clear and figures are well illustrated. I really appreciate the author’s efforts on writing and enjoy reading this paper.     * Proposed algorithm is well designed. I especially like how the proposed approach could be related with the mapping, localization and motion planning procedure in traditional robotics.    * The author considered challenging cases where path is not exactly the same and changes present in the world.     * It shows an interesting, promising, yet unsolved direction to NIPS community, which I believe would draw attentions in short future.    Cons:    * The proposed method is limited to work only with very related paths (with limited perturbed noise or reversed trajectory). And it’s difficult to see the potential of generalize to a very different path using the current feature synthesis algorithm. Human indeed has this capability with very limited exploration.    * Failure cases were not discussed in detail. There are half of the cases failed. Could you analyze more of those cases and point out potential future directions to authors?    * Baselines (especially classic SLAMs) are not favoured under current setting.     * Recurrent settings might be restricted to short paths, e.g. 30 steps in paper’s setting. It is also unclear to me what information hidden states take to policy function. Could you report performance with a non-recurrent policy pi(mu_t, Phi(O_t))?    Some detailed comments:   It seems SLAM baseline suffers a lot in this simple scenario, in which world changes are not very significant. This is quite surprising to me given the capacity of COLMAP. Thus I would like to know more about the author’s experiment on the SLAM baseline to ensure a proper comparison:    * Are you using incremental bundle adjustment or full bundle adjustment setting in COLMAP?     * Do you give BA solver your true camera intrinsic matrix?     * How many frames you used per each path? Still 30? And action between each step is also pure rotation with 30 degree or 40cm forward? Note that both settings are not realistic in real-world applications (images could be received at much higher rate) and could make SLAM based approaches unfavoured.     * Have you actually managed to estimate accurate camera pose and reconstruct dense point cloud with good quality using COLMAP under your setting? Do you think localization failures are due to poor mapping or poor SIFT matching?    * In the path following stage, why do only do SIFT matching against the points only, rather than the full incremental BA pipeline (SIFT matching plus non-linear reprojection error minimization) to estimate the new image of the camera pose?   Robotics community does have a lot of online localization algorithm with loop closure detection capability, e.g. Fabmap. These works should be definitely considered to compare as stronger geometric baselines.