Paper:
Sparse reward is one of the most challenging problems in reinforcement learning (RL). Hindsight Experience Replay (HER) attempts to address this issue by converting a failure experience to a successful one by relabeling the goals. Despite its effectiveness, HER has limited applicability because it lacks a compact and universal goal representation. We present Augmenting experienCe via TeacheR's adviCE (ACTRCE), an efficient reinforcement learning technique that extends the HER framework using natural language as the goal representation. We first analyze the differences among goal representation, and show that ACTRCE can efficiently solve difficult reinforcement learning problems in challenging 3D navigation tasks, whereas HER with non-language goal representation failed to learn. We also show that with language goal representations, the agent can generalize to unseen instructions, and even generalize to instructions with unseen lexicons. We further demonstrate it is crucial to use hindsight advice to solve challenging tasks, but we also found that little amount of hindsight advice is sufficient for the learning to take off, showing the practical aspect of the method.

Review:
Paper Summary: 
The idea of the paper is to improve Hindsight Experience Replay by providing natural language instructions as intermediate goals. 
Paper Strengths:
Unfortunately, there is not many positive points about the paper except that it explores an interesting direction. 
Paper Weaknesses: 
I vote for rejection of the paper due to the following issues:
- It is not clear how a description for a point along the way is provided (when the agent is not at a target). It is not clear how those feedback sentences are generated. That is the main claim of the paper and it is not clear at all.
- The result of DQN is surprising (it is always zero). DQN is not that bad. Probably, there is a bug in the implementation. There should be comments on this in the rebuttal.
- According to several recent works, algorithms like A3C work much better than DQN. Does the proposed method provide improvements over A3C as well?
- The only measure that is reported is success rate. The episode length should be reported as well. I suggest using the SPL metric proposed by Anderson et al. in "On Evaluation of Embodied Navigation Agents".
- Replacing one word with its synonym is considered as zero-shot. That is not really a zero-shot setting. Please refer to the following paper, which is missing in the related work:
Interactive Grounded Language Acquisition and Generalization in a 2D World, ICLR 2018
- The environments are toy environments. The experiments should be carried out in more complex environments such as THOR or House3D that include more semantics.
- What is the difference between this method and providing a large negative reward at a non-target object?
- The paper discusses the advantages of word embeddings over one-hot vectors. That is obvious and not the goal of this paper. 
- It seems the same environment is used for train and test.
------------------------
Post rebuttal comments:
Most of my concerns have been addressed. My new rating is 5. I like the idea of having a compact representation for the hindsight experience replay, but there are still a few issues:
- I expected more complexity in vision and language. I do not agree with the rebuttal that AI2-THOR or House3D are not suitable. This level of complexity would be ok if this paper was among the first ones to explore this domain, but there are already several works. The zero-shot setting (changing the word with its synonym) is also so simplistic.
- The proposed method uses much more annotations than the baselines so the comparisons are not really fair. This information should have been added to the baseline to see how this additional information changes the performance. Basically, it is not clear if the improvement should be attributed to the extra annotation or the way the advice is given.
- The writing is still confusing. For instance, it is mentioned that "Concretely, for each state s ∈ S, we define T as a teacher that gives an advice T(s)", while that is not true since later it is mentioned that "the teacher give advice based solely on the terminal state". These statements are contradictory, and it is not trivial at all to provide an advice for each state.