Paper:
Language creates a compact representation of the world and allows the description of unlimited situations and objectives through compositionality. These properties make it a natural fit to guide the training of interactive agents as it could ease recurrent challenges in Reinforcement Learning such as sample complexity, generalization, or multi-tasking. Yet, it remains an open-problem to relate language and RL in even simple instruction following scenarios. Current methods rely on expert demonstrations, auxiliary losses, or inductive biases in neural architectures. In this paper, we propose an orthogonal approach called Textual Hindsight Experience Replay (THER) that extends the Hindsight Experience Replay approach to the language setting. Whenever the agent does not fulfill its instruction, THER learns to output a new directive that matches the agent trajectory, and it relabels the episode with a positive reward. To do so, THER learns to map a state into an instruction by using past successful trajectories, which removes the need to have external expert interventions to relabel episodes as in vanilla HER.

Review:
This work attempts to learn instruction following agents without the requirement for a paired instruction, behavior corpus and instead simply relying on environment interaction and using a form of hindsight relabeling to learn how to relate language and behavior. 
Introduction: 
It thus leads to the following questions: are we eventually making the rein-forcement learning problem harder, or can we generate learning synergies between policy learningand language acquisition? -> it really doesn’t seem like the point of instruction following is that. It seems like you want instruction following so that you can communicate novel objectives to your agent, with the promise of generalization. Maybe reword?
The motivation for using HER makes sense, but maybe a bit more would be useful to describe why we need text here and not just regular HER. 
I think this line “triggers alearning synergy between language acquisition and policy learning” is pretty confusing and not really adding too much value. Would remove. 
Overall motivation makes sense, do language grounding in an interactive way much more effectively by leveraging hindsight relabeling but to get around the circular problem of hindsight generation leverage a model of successful behaviors seen thus far. This is a pretty neat thing to do!
Textual HER:
“We emphasize that such signals are inherent to the environment, and an external expert does not provide them” -> I do not think this is true. Rewards do not magically show up in the environment, they have to be provided. I get what you’re trying to say but this statement is very often not true. Please revise. 
Conceptual question: what happens if none of the random trajectories are successful coz the reward is so sparse? Wouldn’t this be prohibitive? Importantly, I would be curious to understand how the number of entries in D affects the relabeling function m_omega and how this can be good or bad depending on the schedule of training. For instance if the D is very small at the start, it’s not going to be very good at doing the relabeling and might be erroneous.
Experiments
“Surprisingly, even highly noisy mappers, with a 80%noise-ratio, still provides an improvement over vanilla DQN-agents” -> do you know why?
The synthetic noisiness that you introduce is from a different distribution than the type of noisiness you’d expect from just having very few successful trajectories to train m_omega right? How can we evaluate that?
If we consider the number of successful trajectories obtained just by accident, is it even 1000 or 5000 as required to train the m_omega. If this is a negative feedback loop couldn’t it just keep getting worse because we are using erroneous m. Should we use some notion of uncertainty or something to know when to relabel with m? Or does it happen always
I generally like Section 4.2 -> nicely motivated!
“We emphasize again that it is impossible to have an external expert to apply HER in the general case” -> why??
Can the authors introduce other baselines. For instance the recent paper from Yiding Jiang, Chelsea Finn, Shane Gu and others might be a start. Consider corpus based instruction following would be another. Maybe these can be compared in terms of the number of instructions that need to provided to it? But I think for a successful ICLR paper, we would need 1-2 more meaningful baselines. 
“Finally, we observe a virtuous circle that arises.” -> is there some mechanism to ensure that this is a virtuous cycle and not a vicious one? Couldn’t we just have horrible label corruption and then everything goes bad?
How easily would this scale to more temporally extended tasks in minigrid which have larger grids and more challenging tasks which are harder to solve in the sparse reward case?
Can we analyze whether the language goal space has some favorable generalization properties over a state based goal space as typical HER uses?
The language analysis in Section 4.4 is quite insightful and shows the good performance of the instruction generator over time. 
How would this fare as language got more ambiguous and multimodal and the instruction generator had a harder time as well as HER might generalize more poorly?
Related Work
Fu et al (From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following) might be relevant for instruction following as well, and some of Karthik Narasimhans work. 
Learning interactively with language would also be related to Co-Reyes et al (Guiding Policies with Language via Meta-Learning)
Yiding Jiang’s recent work would also be relevant (https://arxiv.org/abs/1906.07343)
Overall I like the formulation, and it seems pretty useful for instruction following. But we need more comparisons, and a little more motivation on when this thing might become degenerate coz of the m labeling. Perhaps even a discussion/experiment on the uncertainty measure might be helpful.