Paper:
Learning to Navigate in Cities Without a Map
Navigating through unstructured environments is a basic capability of intelligent creatures, and thus is of fundamental interest in the study and development of artificial intelligence. Long-range navigation is a complex cognitive task that relies on developing an internal representation of space, grounded by recognisable landmarks and robust visual processing, that can simultaneously support continuous self-localisation ("I am here") and a representation of the goal ("I am going there"). Building upon recent research that applies deep reinforcement learning to maze navigation problems, we present an end-to-end deep reinforcement learning approach that can be applied on a city scale. Recognising that successful navigation relies on integration of general policies with locale-specific knowledge, we propose a dual pathway architecture that allows locale-specific features to be encapsulated, while still enabling transfer to multiple cities. A key contribution of this paper is an interactive navigation environment that uses Google Street View for its photographic content and worldwide coverage. Our baselines demonstrate that deep reinforcement learning agents can learn to navigate in multiple cities and to traverse to target destinations that may be kilometres away. The project webpage http://streetlearn.cc contains a video summarizing our research and showing the trained agent in diverse city environments and on the transfer task, the form to request the StreetLearn dataset and links to further resources. The StreetLearn environment code is available at https://github.com/deepmind/streetlearn.

Review:
This paper presents an application of a deep reinforcement learning model to a new navigation task based on real-world images taken from Google Street View. The paper is well written and easy to read. The task is interesting, the results are impressive and show that an external memory is not even required in order to achieve a good performance.  My main concerns are that: a) The model cannot work on maps it has not seen during training, and requires to be fine-tuned on new maps. Also, the point of doing transfer between multiple cities does not seem obvious, as the model still needs to be trained (tuned) on the new map and does not perform as well as a single-city model. b) The model is evaluated on the training maps themselves, but using held-out areas. The size of these held-out areas seem to have a significant impact on the model which suggests that the model is overfitting on the training maps.  A couple of comments / questions:  1) I’m not sure I understood how the landmarks are selected to define the goal, even after looking at the Appendix. Are the landmarks fixed and always the same across a city? The example Figure 2. a) shows an example with 5 landmarks used to represent the location of a particular goal, and I initially thought that given a location, g_t was computed using some nearby landmarks. But it turns out that the landmarks you consider are always the same (all of them) and not only the ones nearby?  2) I found line 138 "the agent receives a higher reward for longer journeys" a bit confusing. It sounds like the agent is given more reward for taking paths longer than necessary. It is not totally clear to me why you would want to give more reward for distant goals (in the end you want to find distant goals as much as the nearby ones). Instead I would have found it natural to give a higher reward to the agent if it is able to find the goal quickly.  3) Section 4.2 suggests that curriculum learning is an important part of the training procedure, but Figure 3) c) indicates that ER alone (with the appropriate range) converges faster and to a better final performance. Is that right? Any reason to keep using curriculum learning for this task?  4) I was surprised by how much the auxiliary task of heading is helping the model given the simplicity of this task. Any intuition of why it helps that much? Doesn’t the loss of this task converge to zero very quickly?  5) Figure 5: did you observe that during training, the agent tends to avoid the held-out areas naturally? Or, given a random goal location, it will navigate through it using the held-out areas as much as the other ones? The gap of performance between “fine” and “coarse” grid size is quite significant and I’m wondering if this can be an issue resulting from the agent not being familiar with the held-out areas at all.  6) When the model is trained on some cities and fine-tuned on a new city, how long does it take to fine tune it compared to training a model from scratch on the new city alone? I think this is an important result to know. The experiments show that a single-city network performs better than the ones trained jointly and fine-tuned, so if fine-tuning is not faster, I don’t see the point in training anything else than single-city networks.  7) Line 74 and line 172, Lample & Chaplot should be cited along with Mirowski et al. and Jaderberg et al., as they were the firsts to publish results showing the benefits of training with auxiliary tasks.