Paper:
This work presents an exploration and imitation-learning-based agent capable of state-of-the-art performance in playing text-based computer games. Text-based computer games describe their world to the player through natural language and expect the player to interact with the game using text. These games are of interest as they can be seen as a testbed for language understanding, problem-solving, and language generation by artificial agents. Moreover, they provide a learning environment in which these skills can be acquired through interactions with an environment rather than using fixed corpora. One aspect that makes these games particularly challenging for learning agents is the combinatorially large action space. Existing methods for solving text-based games are limited to games that are either very simple or have an action space restricted to a predetermined set of admissible actions. In this work, we propose to use the exploration approach of Go-Explore (Ecoffet et al., 2019) for solving text-based games. More specifically, in an initial exploration phase, we first extract trajectories with high rewards, after which we train a policy to solve the game by imitating these trajectories. Our experiments show that this approach outperforms existing solutions in solving textbased games, and it is more sample efficient in terms of the number of interactions with the environment. Moreover, we show that the learned policy can generalize better than existing solutions to unseen games without using any restriction on the action space.

Review:
This paper considers the task of training an agent to play text-based computer games. One of the key challenges is the high-dimensional action space in these games, which poses a problem for many current methods. The authors propose to learn an LSTM-based decoder to output the action $a_t$ by greedily prediction one word at a time. They achieve this by training a sequence to sequence model on trajectories collected by running the game using a previously proposed exploration method (Go-Explore). While the results are promising, there might be limited novelty beyond training a sequence to sequence model on pre-collected trajectories. Further, the experiments are missing key elements in terms of proper comparison to baselines. 
Pros:
1. Nice idea for tackling the unbounded action space problem in text-based games. 
Cons:
1. The method depends on the assumption that we can get a set of trajectories with high rewards. This seems a pretty strong assumption. In fact, the authors use a smaller set of admissible actions in order to collect these trajectories in the first place - this seems to not be in line with the goal of solving the large action space problem. If we assume access to this admissible action function, why not just use it directly?
2. Some of the empirical results may not be fair comparisons (unless I'm missing something). For example, all the baselines for the CookingWorld games use $\epsilon$-greedy exploration. Since the Go-Explore method assumes access to extra trajectories at the start, this doesn't seem fair to the other baselines which may not observe the same high-reward trajectories.
Other comments:
1. Do you use the game rewards to train/finetune the seq2seq model or is it only trained in a supervised fashion on the trajectories? (like an imitation learning setup)
2. How critical is the first step of producing high reward trajectories to the overall performance? Some more analysis or discusssion on this would be helpful to disentangle the contribution of GoExplore from the seq2seq action decoder.