Paper:
Recent research efforts enable study for natural language grounded navigation in photo-realistic environments, e.g., following natural language instructions or dialog. However, existing methods tend to overfit training data in seen environments and fail to generalize well in previously unseen environments. In order to close the gap between seen and unseen environments, we aim at learning a generalizable navigation model from two novel perspectives: (1) we introduce a multitask navigation model that can be seamlessly trained on both Vision-Language Navigation (VLN) and Navigation from Dialog History (NDH) tasks, which benefits from richer natural language guidance and effectively transfers knowledge across tasks; (2) we propose to learn environment-agnostic representations for navigation policy that are invariant among environments, thus generalizing better on unseen environments. Extensive experiments show that our environment-agnostic multitask navigation model significantly reduces the performance gap between seen and unseen environments and outperforms the baselines on unseen environments by 16% (relative measure on success rate) on VLN and 120% (goal progress) on NDH, establishing the new state of the art for the NDH task.

Review:
This paper addresses some challenges of following natural language instructions for navigating in visual environments. The main challenge in such tasks is the scarcity of available training data, which results in generalization problems where the agent has difficulty navigating in unseen environments. Therefore, the authors propose two key ideas to tackle this issue and incorporate them in the reinforced cross-modal matching (RCM) model (Wang et al, 2019). First, they use a generalized multitask learning model to transfer knowledge across two different tasks: Vision-Language Navigation (VLN) and Navigation from Dialog History (NDH). This results in learning features that explain both tasks simultaneously and hence generalize better. Moreover, by training on both tasks the effective size of training data is increased significantly. Second, they propose an environment-agnostic learning technique in order to learn invariant representations that are still efficient for navigation. This prevents overfitting to specific visual features of the environment and therefore helps improving generalization. The contribution of this paper is combining and incorporating these two key ideas in the RCM framework and verifying it on VLN and NDH tasks. This approach is novel compared to prior results in tackling the VLN task. Their experimental results show that the two proposed techniques improve generalization in a complementary fashion, measured by decreased performance gap between seen and unseen environments.  They demonstrate that their technique outperforms state-of-the-art methods on unseen data on some evaluation metrics.
Even though the two key ideas proposed in the paper has been explored in the literature in other contexts, this paper contributes to natural language guided navigation by incorporating these ideas in a unified framework and demonstrates promising results on two datasets. Therefore, I would recommend accepting this paper if some issues in delivery and clarity are addressed.
In particular, Section 3, where the authors introduce the novelty of the paper in more detail, could be better explained and a cleaner line should be drawn between prior results from other papers and novel results proposed by this paper. In general, the new ideas would require more emphasis, since they are somewhat lost between adaptations from prior work. Most importantly, Eq. (3) is stated without sufficient motivation and would require a more detailed explanation.
In addition to these points, I would like to disclose some recommendations that might improve the paper but are not strictly part of my decision. In some cases the notation is not clear and some variables are not defined or explained. For instance, after Eq. (8) Attention(.) is used without citation or definition, in Eq. (9) Wc and Wu are not defined and some of the notation in Eq. (6)-(7) are not defined. Moreover, reading the decrease in performance gap from the presented table format is inconvenient and a better visual representation might help demonstrating the improvement in generalization better.  Lastly, I noticed that the navigation error for shared decoder is slightly higher than for separate encoders in Table 3, even though it outperforms the separate encoders in every other measures. Is there a particular explanation for this?