Meta-Reinforcement Learning for Mastering Multiple Skills and Generalizing across Environments in Text-based Games (2021)

Abstract: Text-based games can be used to develop task-oriented text agents for accomplishing tasks with high-level language instructions, which has potential applications in domains such as human-robot interaction. Given a text instruction, reinforcement learning is commonly used to train agents to complete the intended task owing to its convenience of learning policies automatically. However, because of the large space of combinatorial text actions, learning a policy network that generates an action word by word with reinforcement learning is challenging. Recent research works show that imitation learning provides an effective way of training a generation-based policy network. However, trained agents with imitation learning are hard to master a wide spectrum of task types or skills, and it is also difficult for them to generalize to new environments. In this paper, we propose a meta reinforcement learning based method to train text agents through learning-to-explore. In particular, the text agent first explores the environment to gather task-specific information and then adapts the execution policy for solving the task with this information. On the publicly available testbed ALFWorld, we conducted a comparison study with imitation learning and show the superiority of our method.

## introduction a text-based game, such as zork @xcite , is a text-based simulation environment that a player uses text commands to interact with. for example, given the current text description of a game environment, users need to change the environmental state by inputting a text action, and the environment returns a text description of the next environmental state. users have to take text actions to change the environmental state iteratively until an expected final state is achieved @xcite . solving text-based games requires non-trivial natural language understanding/generalization and sequential decision making. developing agents that can play text-based games automatically is promising for enabling task-oriented, language-based human-robot interaction (hri) experience @xcite . supposing that a text agent can reason a given command and generate a sequence of text actions for accomplishing the task, we can then use text as a proxy and connect text inputs and outputs of the agent with multi-modal signals, such as vision and physical actions, to allow a physical robot operate in the physical space @xcite . given a text instruction or goal, reinforcement learning (rl) @xcite is commonly used to train agents to finish the intended task automatically. in general, there are two approaches to train a policy network to obtain the corresponding text action: generation-based methods that generate a text action word by word and choice-based methods that select the optimal action from a list of candidates @xcite . the list of action candidates in a choice-based method may be limited by pre-defined rules and hard to generalize to a new environment. in contrast, generation-based methods can generate more possibilities and potentially have a better generalization ability. therefore, to allow a text agent to fully explore in an environment and obtain best performance, a generation-based method is needed @xcite . however, the combinatorial action space precludes reinforcement learning from working well on a generation-based policy network. recent research shows that imitation learning @xcite provides an effective way to train a generation-based policy network using demonstrations or dense reinforcement signals @xcite . however, it is still difficult for the trained policy to master multiple task types or skills and generalize across environments @xcite . for example, an agent trained on the task type of slicing an apple cannot work on a task of pouring water. such lack of the ability to generalize precludes the agent from working on a real interaction scenario. to achieve real-world hri experience with text agents, two requirements should be fulfilled: 1) a trained agent should master multiple skills simultaneously and work on any task type that it has seen during training; 2) a trained agent should also generalize to unseen environments. meta-reinforcement learning (meta-rl) is a commonly used technique to train an agent that generalizes across multiple tasks through summarizing experience over those tasks. the underlying idea of meta-rl is to incorporate meta-learning into reinforcement learning training, such that the trained agent, e.g., text-based agents, could master multiple skills and generalize across different environments @xcite . in this paper, we propose a meta-rl based method to train text agents through learning-to-explore. in particular, a text agent first explores an environment to gather task-specific information. it then updates the agent's policy towards solving the task with this task-specific information for better generalization performance. on a publicly available testbed, alfworld @xcite , we conducted experiments on all its six task types (i.e., pick & place, examine in light, clean & place, heat & place, cool & place, and pick two & place), where for each task type, there is a set of unique environments sampled from the distribution defined by their task type (see section 5.1 for statistics). results suggest that our method generally masters multiple skills and enables better generalization performance on new environments compared to alfworld @xcite . we provide further analysis and discussion to show the importance of task diversity for meta-rl. the contributions of this paper are: ## method we use neural networks to map observations to actions. given the general setting of metareinforcement learning through learning-to-explore, our method contains three modules: an execution policy neural network π ψ , a task identifier neural network q θ , and an exploration policy neural network p φ , where ψ, θ, and φ denote parameters of the three neural networks, respectively. as shown in figure 1 , an exploration policy p φ is trained to generate a task-specific feature vector z µ , which is then input to an execution policy π ψ for generating actions. during training, a task identifier is used to generate supervised signals z µ of z µ , and is not used during testing. because of z µ , π ψ can adapt quickly and generalize well in a new task. the π ψ , q θ and p φ are all encoder-decoder architectures. for π ψ , it takes a goal g and a k-step roll-out trajectory τ @xmath0, a t-k , o t-k+1 , . . . , a t-1 , o t ) from time t-k+ 1 to time t as inputs, and outputs the current action a t , where o 0 is obtained by executing the "look" action at the beginning. o 0 is used because it is the only observation that lists the different areas of the room. q θ takes a task index µ as an input and outputs the task-specific feature z µ , which is only used during training. p φ takes a goal and a k-step roll-out trajectory τ @xmath1, a t-k , o t-k+1 , . . . , a t-1 , o t ) as inputs and outputs an estimated task-specific feature z µ . our goal is to make an execution policy π(a t |g, τ t ) generalizable across tasks. if we train π using imitation learning, it is critical to have enough training samples of {(g, τ t , a t )} following some distributions to have good generalization performance. but because of the combinatorial complexity of τ t , it is hard to obtain enough data of a t . learning from conditional variational auto-encoder (cvae) @xcite , we factorize π with a task-specific hidden variable z and use z to facili-tate the generation of a t , namely, π(a t |g, τ t ) = @xmath2, τ t )π(a t |z, g, τ t )dz, (1) where z ∼ n (z µ , σ 2 i) is assumed to follow a gaussian distribution, the aforementioned taskspecific feature vector z µ is the mean vector and σ 2 is the variance. during testing, we can then generate actions by first generating a task-specific hidden variable z with p(z|g, τ t ) and then generating the action with π(a t |z, g, τ t ). because z encodes taskspecific features, it helps π generate more proper actions for the current task µ. optimizing (1) amounts to maximise evidence lower bound (elbo) @xcite : ## experiments to demonstrate the generalization ability of our meta-reinforcement learning algorithm across tasks, we conducted a set of experiments with the alfworld platform @xcite . text environments of alfworld are aligned with 3d simulated environments from alfred @xcite , which makes alfworld a good proxy for our human-robot interaction scenario. ## conclusion we study the generalization issue of text-based games, and develop a meta-reinforcement learning method with a learning-to-explore approach. in particular, we first use an exploration policy network to learn a task-specific feature vector, and use this feature vector to help another execution policy network adapt to a new task. to train the exploration and execution policy network, we use a task identifier to embed a task index, and maximize the likelihood of the execution policy network endto-end. to demonstrate the generalization ability of our method, we conducted a set of experiments on the publicly available testbed alfworld. in general, we find that our method has better generalization performance on a wide spectrum of task types and environments. we leave the investigation of explanability, the unbalance problem of task types, and the training speed as the future research directions.