Paper:
Speaker-Follower Models for Vision-and-Language Navigation
Navigation guided by natural language instructions presents a challenging reasoning problem for instruction followers. Natural language instructions typically identify only a few high-level decisions and landmarks rather than complete lowlevel motor behaviors; much of the missing information must be inferred based on perceptual context. In machine learning settings, this is doubly challenging: it is difficult to collect enough annotated data to enable learning of this reasoning process from scratch, and also difficult to implement the reasoning process using generic sequence models. Here we describe an approach to vision-and-language navigation that addresses both these issues with an embedded speaker model. We use this speaker model to (1) synthesize new instructions for data augmentation and to (2) implement pragmatic reasoning, which evaluates how well candidate action sequences explain an instruction. Both steps are supported by a panoramic action space that reflects the granularity of human-generated instructions. Experiments show that all three components of this approach-speaker-driven data augmentation, pragmatic reasoning and panoramic action space-dramatically improve the performance of a baseline instruction follower, more than doubling the success rate over the best existing approach on a standard benchmark.

Review:
This paper builds upon the indoor vision and language-grounded navigation task and sequence-to-sequence model described in (Anderson et al, 2017), by introducing three improvements: 1) An encoder-decoder-like architecture, dubbed “speaker-follower” model, that not only decodes natural language instructions into a sequence of navigation actions using seq2seq, but also decodes a sequence of navigation actions and of image features into a sequence of natural language instructions using a symmetric seq2seq. That speaker model can then be used for scoring candidate routes (i.e., candidate sequences of images and actions) w.r.t. the likelihood of the natural language instruction under the speaker model. This enables a form of planning for the seq2seq-based agent. 2) That same speaker architecture can be used to generate new natural language descriptions for existing or random-walk generated trajectories through the environment, and thus augment data for training the follower. 3) Instead of considering 60deg field-of-view oriented image crops of the panoramic images as observations for the agent, entailing repetitive rotations to align the agent with the target destination, the agent is presented directly with a 360deg view, and uses an attention mechanism to predict where to look at, then compute the probability of going there in the image. The image and motion are decomposed into 12 yaw and 3 pitch angles.  The authors achieve state-of-the-art performance on the task and do a good ablation analysis of the impacts of their 3 improvements, although I would have liked to see navigation attention maps in the appendix as well. Improvement #3 (panoramic action space) is a significant simplification of the task; luckily the NE and SR metrics are unaffected.  It is worth noting that the idea of using language prediction for navigation in a grounded 3D environment has already been introduced by Hermann et al, 2017: "Grounded language learning in a simulated 3D world”.  Minor remark: The abstract is difficult to understand without reading the paper (both the “pragmatic reasoning” “panoramic action space” could be reformulated as model-based planning and visual attention-based navigation in panoramic images).  In summary: Quality: overall well presented paper that contributes to the development of language-grounded visual navigation agents Clarity: good but reformulate the abstract Originality: rather incremental work, missing reference on language generation for language-grounded agent Significance: good, state-of-the-art results on the Matterport dataset  Update: I have read the author's rebuttal and updated my score accordingly from 6 to 7.